Possible solution: x64 audio bug

[Suspiria]

Renegade-X | x64 AUDIO BUG | Possible attack vectors | 21-03-2019
--------------------------------------------------------------------

Spoiler

• Commenting out these three lines in Engine/Config/BaseEngine.ini:
  Suppress=DevAudio
  Suppress=DevAudioVerbose
  Suppress=DevSound
  So that we can properly log.
  And then start the x64 UDK.exe with the flag "-log", for the funz.
  In-game, use STAT AUDIO and STAT LEVELS to monitor memory management.
   
• Preventing the CommonAudioPoolSize mismatch:
  ../Engine/Config/BaseEngine.ini
  ../UDKGame/Config/DefaultEngine.ini
  ../UDKGame/Config/UDKEngine.ini
  The currently set value of 96000000 in DefaultEngine.ini is NOT 96Mb - as suggested by the comments.
  Either set all of them to a power-of-2 value (e.g. 96MB = 100663296), or set them to 128, 64 or even 0.
   
• With the audio shutting down with a small "bang" on my system, it seems to me that the entire subsystem just shuts down.
  Could the problem be in the codec? ::
  
  Search for older/newer versions of the audio codec DLL's, and try replacing them:
  libvorbis_64.dll     (Vorbis Audio Codec)    (current is 1.3.2.12709)
  libvorbisfile_64.dll     (Vorbis Audio Codec)    (current is 1.3.2.12709)
  libogg_64.dll         (Ogg Container Support)    (current is 1.2.2.12709)
  libresample_x64.dll    (Audio Resampler)    (current is 0.1.3.0)
   
• Unreal Engine 4 library grab
  I am currently running UE4's Ogg Vorbis DLL's on RenX-x64. Running butter smooth so far.
  Those are at versions 1.3.2.0 and 1.2.2.0 - the zero's possibly indicating older and/or final stable builds.
   
• Xiph.org
  Complete Ogg Vorbis source code up to 1.3.6.
  This is a whole other ballgame as we're not even sure that the problem is actually in the codec.
  I like to go deep, but not too deep. ..;p
  So let's not go this route, for now.
   
• Bug reproduction
  Let's keep our eyes open for a quicker way of reproducing this bug.
  Can we trigger it somehow?
  I've personally experienced, and heard from others, that it usually happens at the end of a match; just when the next map starts loading.
  Is there anyone here that has actually encountered this midmatch? Specific maps or servers? Influenced by playercount?
  This kind of information could prove useful and should be carefully documented if we are to work this issue.
   

On 3/21/2019 at 8:41 PM, SonnyX said:

Some members of the developer team have already stated before that these are confirmed bugs and that some have already tried looking into fixing it, yet to my understanding the problem lies in the engine's source-code

Thank you for your input.

I haven't been able to find any information whatsoever about this particular "confirmed bug" with audio in UDK, at all.
Not on the internet anyway.

If the fault is truly in the engine itself, then all the applications in the world that are using UE3/UDK should suffer the exact same fate.
Those are 617+ games.... All having a 10% chance for all audio to go out with a bang, until a hard reset. .....Yet noone is writing about it?!

This makes me wonder where these "bug confirmations" originated from.
And more importantly: how exactly this conclusion was reached.

What I mean to say with that, is that we might not be able to fix this in the source code.
But we may very well come to more thoroughly understand the bug, and be able to work our way around it.

TEST RESULTS | 22-24 march 2019
-----------------------------------

Spoiler

Tested with UE4's Ogg Vorbis libraries. Actually works butter smooth, but still crashes our audio.
Tested with 100663296 instead of 96000000 Bytes CommonAudioPoolSize. No joy.

Now testing with CommonAudioPoolSize 0, disabling the persistent pool for audio data.
So far I've played 32+ games in a row, with no audio shutdown when loading the next map.
Nor have I had any stuttering, or whatsoever.

If you were to Google "CommonAudioPoolSize", you would find that noone, absolutely noone, sets this value higher than between 0 and 512.
Let alone 96000000. I am quoting: "It could fix or create audio issues, so keep that in mind and change it back to 0 if needed."
So,.. 96000000 seems like a ri-di-cul-ous value, to me. And I strongly suspect this to be our causal agent.

It seems waaay too easy though, just .iniïng ourselves out of this mess.
....But it wouldn't be the first time us humans overlook something. ;-D
We might just catch a break here.

Feels really great not to have the 32-bit "Ran out of video memory" crashes anymore.
Tbh, I haven't noticed anything negative at all with the x64 now that the sound issue seems to have been fixed for me.
That leaves me wondering what exactly those other issues with this client are about.

CONCLUSION | 24-03-2019
---------------------------

The x64 audio issue seems to be resolved on my computer.
The following is what I did:

Set CommonAudioPoolSize to 0 in:
../UDKGame/Config/DefaultEngine.ini

Spoiler

@dev: I believe this setting was designed for the Xbox platform. It may be better to keep it at 0 on Windows!

And that's it!

Please do confirm and feedback this solution by replying in this topic or by PM.
Thank you for giving me the opportunity to assist with this.

DH out

Edited March 26, 2019 by DugeHick

[HIHIHI]

I can be another tester forthis. I'll follow your suggestion and set the CommonAudioPoolSize to 0, and try to remember to use 64-bit all the time. When using the 64-bit binary, I do get the soundbug pretty often, so we'll see what happens.

Thanks,

    HIHIHI

[Suspiria]

31 minutes ago, HIHIHI said:

I can be another tester forthis. I'll follow your suggestion and set the CommonAudioPoolSize to 0, and try to remember to use 64-bit all the time. When using the 64-bit binary, I do get the soundbug pretty often, so we'll see what happens.

Thanks,

    HIHIHI

Thank you!
Hope this works out for you too.

If this works for everyone, then Totem Arts might be one step closer to officially supporting x64.
That is, from the launcher.

I replaced my launcher shortcut with the 64-bit UDK.exe, and I've also added -nomoviestartup  to its command line in properties.
With that, it skips the movies and I can start playing almost as quickly as with the launcher.

[SonnyX]

I'm impressed, well documented.

I'll see if I can get this into the next patch if it doesn't seem to cause any negative side-effects. The new launcher will no matter what give a 64-bit option and most likely it will be set to 64-bit by default.

Also about my quote: the thing that I called a confirmed bug was both "Running out of video memory" and "audio bug", somebody seemed to be refuting this, I wasn't saying that both bugs are caused by the UDK engine, I did however say that it was suspected to be in the engine.

[HIHIHI]

I must regretfully report that this is not a full fix for the sound bug, although it may reduce the chance of its occurrence (I cannot say this with certainty as I have a small sample size.). Today, on the official server, after a bot war on Field X, I got the sound bug on Under. : (

HIHIHI

[Suspiria]

On 3/25/2019 at 4:05 AM, HIHIHI said:

I must regretfully report that this is not a full fix for the sound bug, although it may reduce the chance of its occurrence (I cannot say this with certainty as I have a small sample size.). Today, on the official server, after a bot war on Field X, I got the sound bug on Under. : (

HIHIHI

Hi there

That sounds like bad news.
Thank you so much for reporting in though!

Could you check under all these files:
../Engine/Config/BaseEngine.ini
../UDKGame/Config/DefaultEngine.ini
../UDKGame/Config/UDKEngine.ini

If all 7 entries "CommonAudioPoolSize" have indeed been set to 0 ?
Could you perhaps provide me with a log file? If yes read the spoiler.

Spoiler

After doublechecking all 7 CommonAudioPoolSize entries, open up the ../Engine/Config/BaseEngine.ini again.

And comment out these lines:
Suppress=Dev
Suppress=DevAssetDataBase
Suppress=DevAudio
Suppress=DevAudioVerbose
Suppress=DevSound
Suppress=DevStreaming

By putting a semicolon ; before the line.

Logs are kept in ../UDKGame/Logs.
The most recent log is called "launch.log". The others are datetime-stamped.

Now, the best thing would be for the bug to occur again.
And to send me the fresh launch.log then.

If that takes too long, you can also just start your x64 client with the settings, join a game, then exit.
And there will also be some useful information for me in there that will help.

Just not as useful as the crashlog itself.
Do as you like. And thank you.

I'm still going strong with a clean 32+ matches without the issue.
A serious new record for me.

Even if the solution turns out not to work, the logic of my attack vector post still applies.
And I don't plan on giving up just yet.

Could get hard though.
To debug a bug that I can't reproduce on my system anymore. XD

I pretty much documented all my steps here, so if my system goes 50+ things are gonna get tough.

Edited March 26, 2019 by DugeHick

[HIHIHI]

Well... I'm not sure now that they were all set to 0. I only know about two of them... Now I have confirmed that they are all set to 0, though. I have also commented out the lines, and will try to remember to send you the log if the sound bug occurs again.

HIHIHI

EDIT: Whoops, I did that stuff in the SDK...  I'll do it in the game now... : )

Edited March 26, 2019 by HIHIHI

[Suspiria]

11 hours ago, HIHIHI said:

Well... I'm not sure now that they were all set to 0. I only know about two of them... Now I have confirmed that they are all set to 0, though. I have also commented out the lines, and will try to remember to send you the log if the sound bug occurs again.

HIHIHI

EDIT: Whoops, I did that stuff in the SDK...  I'll do it in the game now... : )

I KNEW IT
I had this in my head: what if he's working in his SDK
Hahah
I deleted my other posts in all channels because I thought the fix turned out bad, lol
I hate you
So please do let me know when you pass the 20+, and all is good in my world again
And still thank you for helping with testing

Edited March 26, 2019 by DugeHick

[roweboat]

If more testing needed I can try doing this later in the week.

[Suspiria]

1 hour ago, roweboat said:

If more testing needed I can try doing this later in the week.

Hi Roweboat,

Great!
Please don't feel obligated to read everything in those posts.

If you want to help test this, the only thing that should be required is this:
Set CommonAudioPoolSize to 0 in:
../UDKGame/Config/DefaultEngine.ini

And then just make sure to launch the UDK.exe from the Win64 folder.
If you want, you can set it to skip movies by making a shortcut and then adding -nomoviestartup to the "Target" box. With a space between.

Let's squash this bug guys.

[NodSaibot]

Unless this one is special, you would also have to change it in UDKEngine.ini, as the DefaultEngine.ini should only be used to generate the UDKEngine. It is possible that it generates a new one when you open it, but I can't test that now.

[Suspiria]

41 minutes ago, Sarah. said:

Unless this one is special, you would also have to change it in UDKEngine.ini, as the DefaultEngine.ini should only be used to generate the UDKEngine. It is possible that it generates a new one when you open it, but I can't test that now.

Thank you for chiming in.

Yes I've checked this, and the UDKEngine.ini copies and overwrites from DefaultEngine.ini at startup every time, reliably.
And in BaseEngine.ini, the values are already set to 0 by default, coming from the installer package.

Thank you very much for pointing this out though, as this could have been a major fuckup.

Edited March 26, 2019 by DugeHick

[limsup]

Not working for me. As usual, I get the sound bug immediately on loading in the next map.

[SonnyX]

1 hour ago, limsup said:

Not working for me. As usual, I get the sound bug immediately on loading in the next map.

Could you provide some log files of when that happens?

On 3/25/2019 at 7:56 AM, DugeHick said:

Could you perhaps provide me with a log file? If yes read the spoiler.

  Hide contents

After doublechecking all 7 CommonAudioPoolSize entries, open up the ../Engine/Config/BaseEngine.ini again.

And comment out these lines:
Suppress=Dev
Suppress=DevAssetDataBase
Suppress=DevAudio
Suppress=DevAudioVerbose
Suppress=DevSound
Suppress=DevStreaming

By putting a semicolon ; before the line.

Logs are kept in ../UDKGame/Logs.
The most recent log is called "launch.log". The others are datetime-stamped.

Now, the best thing would be for the bug to occur again.
And to send me the fresh launch.log then.

If that takes too long, you can also just start your x64 client with the settings, join a game, then exit.
And there will also be some useful information for me in there that will help.

Just not as useful as the crashlog itself.
Do as you like. And thank you.

 

 

[Suspiria]

2 hours ago, limsup said:

Not working for me. As usual, I get the sound bug immediately on loading in the next map.

Hmm interesting.

Are you sure you're not working inside some SDK?
Did you save the .ini file(s) while UDK was NOT running?

I'm still going strong with 35+ while before I had it like once every ~7-10 games.
When I reach 50+ I'm gonna conclude that *something* permanently fixed the issue on my system.

Looking forward to HIHIHI's new results now that we know he's edited the right file.

[limsup]

10 hours ago, DugeHick said:

Are you sure you're not working inside some SDK?

I was using the game's configs with the game's binary.

10 hours ago, DugeHick said:

Did you save the .ini file(s) while UDK was NOT running?

That goes without saying.

[roweboat]

7 hours ago, limsup said:

 

That goes without saying.

 

no stone should be left unturned when debugging =D

[Suspiria]

7 hours ago, limsup said:

That goes without saying.

I'm sure I seem patronizing here.

The reason I'm saying this is because UDK will rewrite certain .INI's on exit.
That is: With the (old) values that it originally started up with.

If one were to change those .INI's while the client is running, and then think that it will be enough to just restart the client.......
...Then one would actually still be working with the old value(s), because the engine has changed that back on exit. **

So, even if one is a really smart person - and I do appreciate your intelligence - :
One may simply not expect this particular thing to happen.

And we will receive another report that the fix ain't working.
And - this goes without saying - we would all lose our valuable time and energy on figuring thát one out.  

I want to make sure that doesn't happen.

** Except for the Default .ini's. Those particular .ini's actually do not get overwritten on exit.
So now that you mention it: It wouldn't even be a problem to change it while it's running, as we are working with DefaultEngine.ini here.

[Suspiria]

I am sorry to report that after 49 successful games, I have now encountered the bug again.

Therefore, I hereby nominate this bug for PITA-bug of the year.
We'll have to take this to the next level.

I still recommend setting that value to 0 though.
As it's not supposed to be non-zero at all. (Xbox only)

I'd like to ask everyone to keep your eyes and ears open.
For a pattern, or a way of reproduction. Anything at all.

That would help.

It's still pretty odd imo that there are so many games made with UDK and UE3 and nobody is complaining except RenX.
Or so it seems anyway.

[HIHIHI]

I have also now encountered it - although I only played about ten or eleven games without it. I double-checked all the "commonaudiopoolsize" entries in my install of the game, and they were all still set to 0.

Here is my launch.log from after I got the soundbug - it happened when Under started. (I played a bit during Under before I stopped, so it may be a ways back.)

Launch.log

HIHIHI

[yosh56]

So I was getting this every game the last week, but I used this 'not-quite-fix' and so far haven't gotten one. I think it's kind of a bandaid for a larger problem, but it's still seeming to be 'better'. 

Can continue to look into it though. Good stuff.

[Suspiria]

The function allocates persistent memory for the XMA audio system of the Xbox 360.
It may not fix our issue, yet it doesn't seem to be supposed to be set non-zero on PC.

With a in-file search application, I've found that this value we set is actually cooked into maps aswell.

Therefore I still recommend setting this value to 0 in all ini's, in the next cook.
To rule out the possibility that the map itself somehow overrides the .ini's.

In the meantime I'll be on the lookout for a pattern, and dive deeper into the belly of the beast.

[Suspiria]

On 3/26/2019 at 5:57 PM, Sarah. said:

Unless this one is special, you would also have to change it in UDKEngine.ini, as the DefaultEngine.ini should only be used to generate the UDKEngine. It is possible that it generates a new one when you open it, but I can't test that now.

 

On 3/26/2019 at 6:04 PM, DugeHick said:

Yes I've checked this, and the UDKEngine.ini copies and overwrites from DefaultEngine.ini at startup every time, reliably.
Thank you very much for pointing this out though, as this could have been a major fuckup.

Ahhh

I had indeed quadruple-checked this!! Only now I've discovered that the engine can be horribly inconsistent.
Sometimes it overwrites the existing UDK ini's, other times it doesn't. It really depends on the weather outside.
I should never have used the word "reliably" there.

Thanks again for pointing this out, Sarah.

On 4/7/2019 at 2:05 AM, yosh56 said:

So I was getting this every game the last week, but I used this 'not-quite-fix' and so far haven't gotten one. I think it's kind of a bandaid for a larger problem, but it's still seeming to be 'better'. 

Can continue to look into it though. Good stuff.

Cool!
Yes, I concur that there's still a reasonable chance that this may be a 'bandaid'.

Curious fact remains, that I can't find any complaints on the internet about this particular kind of bug in other games made with UDK. Or UE3 for that matter. It seems waaay too RenX specific to me, to really consider it to be permanently out of our reach.

If anyone is able to find exactly this bug online with other UDK games, surely do let me know.
I had no success with it.

~~

And thank you for enabling the little 64-bit option in the launcher, Yosh & #dev !
I'm really happy with that, and so are some others I believe. 

Have a nice day!

- DH

Edited April 13, 2019 by DugeHick

[Guest Gliven]

Borderlands 2 which runs on UE3 had a sound bug, but i dont know if its related. I've run into it twice myself. It took me hours of searching to find the fix. But i believe this was the one that worked.

Is there a setting in the engine files similar to this one in the video?
 

There was about 3 or 4 "solutions" out there. Can't believe i missed the most obvious fix in the launcher.

Might be worth it to look into the other problems people had with the game's sound

Edited April 13, 2019 by Gliven

[Suspiria]

3 hours ago, Gliven said:

Borderlands 2 which runs on UE3 had a sound bug, but i dont know if its related. I've run into it twice myself. It took me hours of searching to find the fix. But i believe this was the one that worked.

Is there a setting in the engine files similar to this one in the video?

There was about 3 or 4 "solutions" out there. Can't believe i missed the most obvious fix in the launcher.

Might be worth it to look into the other problems people had with the game's sound

Heya! Thank you for sharing that, sir!

That is a mute issue. And doesn't look like the same sound bug to me.

There are certainly many games with "audio issues" out there, but it's important here to discern between the kinds of issues.
Ye have mute situations, user errors, sound quality issues, wrongly selected sound device, or simply no output at all even though nothing is set to muted.
Usually it's one of those categories.

During my research, though, I have found that in our case there is actually a signal being outputted to the buffer when audio goes out.
If you look at your VU-meter in Windows when it happens, you'll find that something is actually overloading that system.
You can also hear it when you change the volume. It's definately *doing something*.

It looks to me to be some kind of algorithm being stuck in a loop overloading the buffer with somewhat of a flat signal - can't call it a wave at all.
As if its being fed corrupted memory, and the kernel going into a protection mechanism. I've never seen anything like it.

I've also noticed that the audio from other programs sounds somewhat less in quality as long as I leave UDK.exe running like that. 
Meaning that it's definately still throughputting *something* into the sound device.

We have a very, very particular kind of situation, that I haven't been able to find on the webs with any other UDK-based game.
And which, to my knowledge, does not fall in the usual category of audio issues.

That's what still makes me think that WE are doing something in the code - or in a setting - that is actually causing it.
And this is why I'm sharply on the lookout for any kind of abnormal setting, user pattern, or quirk in our code that might cause it.

Thanks again for sharing your Borderlands 2 experiences. It might just have been a big, big help. Ye never know with these things.
Like roweboat said: No stone should be left unturned.

I don't believe it's the same issue though.
As a mute issue should simply zero the signal, and not go out with a small bang while showering your audio device with a bunch of bullshit-bytes.

-DH

[RypeL]

Maybe stat audio could show something interesting when this bug happens: https://api.unrealengine.com/udk/Three/StatsDescriptions.html#STAT AUDIO

[Suspiria]

11 hours ago, RypeL said:

Maybe stat audio could show something interesting when this bug happens: https://api.unrealengine.com/udk/Three/StatsDescriptions.html#STAT AUDIO

Quote

Commenting out these three lines in Engine/Config/BaseEngine.ini:
Suppress=DevAudio
Suppress=DevAudioVerbose
Suppress=DevSound
So that we can properly log.
In-game, use STAT AUDIO and STAT LEVELS to monitor memory management.

Yes RypeL, looked into that too.

There seemed to be nothing out of the ordinary there.
It's been a while but if I remember correctly I've also caught the bug while looking at STAT AUDIO.
And there was absolutely nothing to see changing there. Nada.

If I could look into that again, for the sakes of human-memory inaccuracy, I would.
Fact is though , that the bug barely happens to me anymore now.
It hasn't since the last ~25 games, and before that the record was 49 games without it.

With this frequency, it's gonna be more difficult for me to help research this.
Except for 3rd party logs - which do not show STAT AUDIO.

It would really help to find a way to reproduce this issue.
To find the pattern.

[roweboat]

Ahhh, that makes sense. Every time I have the audio bug, I hear my speakers "pop" (something these speakers don't do when normally turning on and off)

[Suspiria]

Hey guys how you doing
Happy new year

I've managed to reproduce what seems like exàctly "the" sound bug, with a 90% frequency.

Spoiler

The bug occurs a couple of seconds after the client receives travel instructions from the server.
Within those couple of seconds, the server is non-seamlessly switching maps and is therefore not yet ready to receive the client.
Then àfter those couple of seconds, a normal routine-looking GC sweep is kicked off on the client that cleans out a small part of objects. And a couple hundred of milliseconds after thàt: audio goes out.

With this particular sweep, the current map('s FoV) is unloaded and we are now served what looks like the watermark-background of the mapvoting screen (?): a darkish 2D background with the C&C faction logo's.
And that moment is exactly the point in time where the sound bug is triggered.
That is, the one that I've been reproducing here.

This particular realtime sweep does NOT seem to be applied in this fashion when the server is fast enough to receive the client before it's time for that sweep. Depending on the server's map-switching speed, being non-seamless and all.
Hence the bug not consistently occurring.

Rather than in the GC itself, I suspect the actual fault to occur in how the native ClientTravel functions relate to Windows 10's memory management conventions. Almost as if some deallocator, along with the map, also accidentally marks audio-related memory to be GC'd or zero'd, resulting in this typical flat-line signal being outputted. And one would only experience such at the point where the garbage collector actually goes to free this (mis)marked territory.

So that could be why the bug kicks in at the point of the normal routine GC sweep.
It's a bit of a guess though.

It all looks like a problem with memory management, on Windows 10 specifically.
As I have eradicated the sound bug in my reproduction setup by setting the x64 binary's compatibility mode to Windows Vista.

Do this now please.

UDK was never designed for Windows 10.
It is officially unsupported.

So as far as I can see, it pretty damn well looks like this could be our true fix.

Today is my birthday, so this possible solution #2 is my treat to Renegade-X today. Hope it works!

Edited July 13, 2020 by DugeHick
change "anything but Windows 10" to specifically Windows Vista

[Suspiria]

 

[Reivax]

i will try that DugeHick  thanks  
happy birthday  

[Agent]

2 hours ago, DugeHick said:

Hey guys how you doing
Happy new year

I've managed to reproduce what seems like exàctly "the" sound bug, with a 90% frequency.

  Reveal hidden contents

The bug occurs a couple of seconds after the client receives travel instructions from the server.
Within those couple of seconds, the server is non-seamlessly switching maps and is therefore not yet ready to receive the client.
Then àfter those couple of seconds, a normal routine-looking GC sweep kicks in that cleans out a small part of objects.

With this particular sweep, the current map('s FoV) is unloaded and we are now served what looks like a transition map; a darkish 2D background with the C&C faction logo's.

And thát, is exàctly the point in time where the sound bug is triggered.
That is, the one that I've been reproducing here.

This particular realtime sweep does NOT seem to be applied in this fashion when the server is fast enough to receive the client before it's time for that sweep. Depending on the server's map-switching speed, being non-seamless and all.

Hence the bug not consistently occurring.

Rather than in the GC itself, I suspect the actual fault to occur in how the native ClientTravel functions relate to Windows 10's memory management conventions. Almost as if some deallocator, along with the map, also accidentally marks (X)Audio(2)-related memory to be GC'd or zero'd, resulting in this typical flat-line signal being outputted and/or (by) a crashed algo. And one would only experience such at the point where the garbage collector actually goes to free this (mis)marked territory.

So that could be why the bug kicks in at the point of the normal routine GC sweep.
Which is initiated after a seemingly "random" couple of seconds post non-seamless client-travel initiation. (but before actually travelling)

It's a bit of a wild guess though.

As I have also seen comments by Epic themselves about XAudio2 crashing on them for unknown reasons.
Like an inner exception handler that " sometimes (?) " crashes XAudio2 when handling assertions.
Yes, with that actual question mark written in the comment.

It all looks like a problem with memory management, on Windows 10 specifically.
As I have eradicated the sound bug in my reproduction setup by setting the x64 binary's compatibility mode to anything but Windows 10.

Do this now please.

UDK was never designed for Windows 10.
It is officially unsupported.

So as far as I can see, it pretty damn well looks like this could be our true fix.

Today is my birthday, so this possible solution #2 is my treat to Renegade-X today. Hope it works!

If this proves to be a working solution, or if it doesn't, please reply to this thread with the result. If the issue proves to be resolved with the compatibility setting, perhaps we can auto-set this via the launcher.

[HIHIHI]

I am sad to report that today, with the 64-bit client running in Windows 7 compatibility mode, I got the soundbug. : (

These were the compatibility settings as verified immediately after closing the game, which I did after I verified that the soundbug had indeed occurred.

Played on the new Forest V2.4.

Heard the POP of the soundbug.

Loaded into the main menu (since I did not have the next map downloaded), and had no sound.

Loaded Cliffside in skirmish and had no sound.

F5 exit, take screenshot of compatibility settings.

Windows 10 Home, i5-7300HQ, Realtek audio, I believe.

For what it's worth, I do get the soundbug much less often than I did on my old machine (the specs of which have been posted in the consolidated soundbug topic).

Will send my launch.log to interested parties if desired.

[SonnyX]

12 hours ago, HIHIHI said:

I am sad to report that today, with the 64-bit client running in Windows 7 compatibility mode, I got the soundbug. : (

Can you try with Windows Vista compatibility mode instead?

[HIHIHI]

I will do so.

[Suspiria]

Hello HIHIHI,

I couldn't find your Windows version in your specs post.
Also, when verifying the setting immediately after closing, are you 1000% positive that you set and verified the correct UDK.exe, that is the one in Win64 folder?

I have been completely unable to suffer the soundbug when setting the compatibility option to Windows 8.
I have tried everything to drive it crazy:

• aggressive alt-tabbing
• restart server and then shut it down so the client gets stuck midtravel
• change map while the maps are deleted, etc.

The latter two usually all crash the audio with the pop; with a 80% certainty.
Except for a normal healthy map change; there it's more like 5%, exactly like we know our sound bug.

But with that compatibility setting set at Windows 8, in NO way I can get this bug to occur. Whatever I throw at it.
When it does "stutter" for a bit because of an alt-tab or the sorts, the engine just picks up again and sound gently continues.
Instead of blasting itself out with the boom.

The difference is like night and day on my computer.
So my hopes aren't completely destroyed yet, but your findings are definately scaring me.

Please double-check if you've been setting the right binary in Win64.

... and if you're already playing on Windows 8 then we may be fooked because that OS *should*'ve been fully supported by Epic.

As Windows 8 was released on october 26th, 2012.
And UE4 was released on september 4th, 2014...

Looking forward to hear from you, bracing myself for failure of possible solution #2.

Edited January 16, 2020 by DugeHick

[HIHIHI]

Windows 10 Home is my OS edition. I was in compatibility for Windows 7.

I have just now verified again that I am applying the compatibility settings to the 64-bit binary and that my launcher is set to use 64-bit in settings.

As previously posted, I have switched to Vista compatibility mode (no SP2 or SP3, just plain Vista) as per Sonny's request. However, I have not played enough to reasonably say that my soundbug situation has improved/completely fixed - I get the soundbug but rarely anyway.

Also as reported (at least on Discord), the soundbug has been observed in standard Windows 7 environments before - it is not exclusive to Windows 10.

Another thanks for all your hard work - I am sorry if I must be the bearer of bad news... But hopefully I am at least of some help.

HIHIHI

[Suspiria]

1 hour ago, HIHIHI said:

Windows 10 Home is my OS edition. I was in compatibility for Windows 7.

I have just now verified again that I am applying the compatibility settings to the 64-bit binary and that my launcher is set to use 64-bit in settings.

As previously posted, I have switched to Vista compatibility mode (no SP2 or SP3, just plain Vista) as per Sonny's request. However, I have not played enough to reasonably say that my soundbug situation has improved/completely fixed - I get the soundbug but rarely anyway.

Also as reported (at least on Discord), the soundbug has been observed in standard Windows 7 environments before - it is not exclusive to Windows 10.

Another thanks for all your hard work - I am sorry if I must be the bearer of bad news... But hopefully I am at least of some help.

HIHIHI

I've been looking through all these people's specs and most people did not include OS and the ones who did were at Windows 10.
If I had known people on Windows 7 get it too, I might've done things differently.

Thank you for being the bearer of bad news.
It's not so bad when you bring it.
And you are definately helping.

And it doesn't mean we aren't on to something.

I do believe the bug I'm reproducing is the same bug.
And 50% of the difficulty of a bughunt is to reproduce and sort of isolate the thing.

Spoiler

The quickest way to reproduce the bug is the following:
1. Run a server
2. Launch your Win64 binary without the compatibility setting (!) (if you want the bug)
3. Join the server with the client
4. Use ingame vote to Restart the map
5. Now close the server while it's restarting
6. Get back to your client immediately, and wait 10-30 secs
7. Voila, 80% of the times audio is now permanently gone until the binary is restarted

If you somehow can't exit to main menu (e.g. screen is all black) then you can use something like "open CNC-Lakeside" in your console, to check if audio is gone. Or you can wait like a full minute and it will automatically put you back on the main menu.

So we might just be "over the hill" here.
 

 

Edited January 16, 2020 by DugeHick

[Suspiria]

Small update

So, personally I haven't encountered the bug anymore lately, but we now know that that may not mean anything.
There is a reasonable chance though, that with this new information and reproducibility someone at some point will figure out a work-around.

To kick it off, I have the following candidates in mind:

1. Enabling seamless server travel.
I know there is probably a good reason for this to be hardcoded to FALSE.
But the sound bug really does occur most often at the point of non-seamlessly unloading the map.
So should it be feasible in some way this is definately worth a try.

2. Tweaking servers to load new maps faster.
The client only GC's the map after quite a few seconds of having to wait for the server to accept new connections.
If the server opens up within that timeframe, that might just fix it.
However I'm not exactly sure how to achieve this other than by upgrading hardware.

Closing this one down in my mind for now, and we'll see what the future brings.

-DH

[NodSaibot]

13 hours ago, DugeHick said:

Small update

So, personally I haven't encountered the bug anymore lately, but we now know that that may not mean anything.
There is a reasonable chance though, that with this new information and reproducibility someone at some point will figure out a work-around.

To kick it off, I have the following candidates in mind:

1. Enabling seamless server travel.
I know there is probably a good reason for this to be hardcoded to FALSE.
But the sound bug really does occur most often at the point of non-seamlessly unloading the map.
So should it be feasible in some way this is definately worth a try.

2. Tweaking servers to load new maps faster.
The client only GC's the map after quite a few seconds of having to wait for the server to accept new connections.
If the server opens up within that timeframe, that might just fix it.
However I'm not exactly sure how to achieve this other than by upgrading hardware.

Closing this one down in my mind for now, and we'll see what the future brings.

-DH

I think both of those options are unachievable or not worth the effort or money. If I recall correctly, it was discovered that seamless was the reason for a lot of crashes between maps. And there's not really much faster we can go, personally I host the AllNoobs renx server on ramdisk and the process has complete access to cpu, there's only so fast that it will get. I think we are pretty much at that peak unless we invest 100+$ more into our monthly bills.

[Suspiria]

On 1/14/2020 at 10:44 AM, SonnyX said:

Can you try with Windows Vista compatibility mode instead?

OK. So while the Vista compatibility mode does eliminate the bug on my repro,...
...and I also not once had the bug again personally on retail...  ...it does not seem to be a universal fix.

As at least one person has now explicitly reported having encountered the bug once in Vista mode.
And others were talking about a "reduction", meaning the bug still occurs.

I therefore assume this is not the fix we're truly looking for.
So that's good to know. Onwards....

On 3/8/2020 at 7:34 AM, StoneD said:

I think both of those options are unachievable or not worth the effort or money. If I recall correctly, it was discovered that seamless was the reason for a lot of crashes between maps. And there's not really much faster we can go, personally I host the AllNoobs renx server on ramdisk and the process has complete access to cpu, there's only so fast that it will get. I think we are pretty much at that peak unless we invest 100+$ more into our monthly bills.

Yes I hear you.
So, I've continued tracking down the bug again.  

Edited July 19, 2020 by DugeHick

[Suspiria]

I've been further validating my reproduction setup:

• indeed I cannot reproduce it with the 32-bit client
• symptoms match up to the tiniest detail

This isn't watertight, but it does make it very likely that I'm reproducing the exact same bug.
I mean, that's a start, right?  (joking. this is actually the most important part imo.)

So here is my new research on the causal level:

Spoiler

I've temporally and logically isolated the bug.
With that I mean both "(milli)seconds after/before" and "after/before script statement(s)".

From there I've arrived at the following conclusion which I will explain as good as I can:

The server tells the client: "Hey bro, your plane is about to depart. Start packing your stuff and take out the last garbage."
And because of this message, the client then sort of expects the server to travel NOW, not tomorrow, and starts cleaning up.

ServerTravel -> ProcessServerTravel -> ProcessClientTravel -> PreClientTravel+ClientTravel -> (NATIVE) PlayerController::ClientTravel -> ... -> Browse()

It is at that point of/after Browse(), that the x64-client seems to prematurely mark the current map/datastores as "allowed to be garbage-collected" for some reason. Then after an arbitrary amount of seconds, a routine (periodic) GC sweep kicks in and does the dirty.

The problem with this is that there is a holy rule in UE that there MUST always be some kind of a map loaded.
Hence they invented seamless travel, using a so-called transition map such as the empty, pitch-black map EnvyEntry.udk
Which we have disabled because of afaik were major issues. (crashes?)

"The reason the transition map exists, is that there must always be a world loaded (which holds the map), so we can't free the old map before loading the new one." (Source)

So it makes sense that (part of) our engine crashes if the current map is prematurely GC'd.

It's a little bit similar to prematurely clearing out the last food from your refrigerator the last night before going on holiday, only to find that you really still needed that as breakfast before moving to the airport.

And this is the suggested workaround:

Spoiler

Now, what I've found is that the current map is NOT garbage-collected prematurely if the client ISN'T being informed explicitly of the travel. Therefore the sound-bug is NOT produced in this particular situation.

To do this, in ProcessServerTravel (Rx_Game.uc), add the following and also comment out the following.

So that it looks like this:

    foreach WorldInfo.AllControllers(class'PlayerController', P)
    {
        if ( NetConnection(P.Player) == None )
            LocalPlayer = P;
    }
//    // Notify clients we're switching level and give them time to receive.
//    if (TravelURL != "") {
//        LocalPlayer = ProcessClientTravel(TravelURL, NextMapGuid, false, bAbsolute);
//    }
//    else {
//        LocalPlayer = ProcessClientTravel(URL, NextMapGuid, bSeamless, bAbsolute);
//    }

What will happen now, is that the client won't know what hit 'em when the server switches maps.
But! In Rx_GameViewportClient, it does get continuously (every tick) notified of a connection-error in NotifyConnectionError.

In it's current state, the latter client-side error-function would immediately do a ConsoleCommand("DISCONNECT") to return the client to the main menu.  ....but we could make it so, that it first attempts to reconnect for a while before eventually returning.

Unfortunately, simply replacing ConsoleCommand("DISCONNECT") with ConsoleCommand("RECONNECT") here is not enough. As in that case it will still end up prematurely calling Browse() at the very first sign of the server switching, thereby dooming the client. Since the continuous calling of NotifyConnectionError-call already starts as soon as the server closes the client connections pre-travel.

Therefore we need to devise a workaround that delays that reconnect-procedure client-wise, up until the moment that the server is ready to immediately accept the new connection.

Had Rx_GameViewportClient been a descendant of Actor, then perhaps one would immediately think of a timer-function. But in this case we can store WorldInfo.TimeSeconds upon receiving the first call to NotifyConnectionError, and from there on with each subsequent call check if enough time has elapsed yet to initiate the reconnect.

Here are the code adjustments for Rx_GameViewportClient.uc:

Spoiler

var int FirstConnError_TimeSeconds;

function NotifyConnectionError(EProgressMessageType MessageType, optional string Message=Localize("Errors", "ConnectionFailed", "Engine"), optional string Title=Localize("Errors", "ConnectionFailed_Title", "Engine") )
{
    local WorldInfo WI;

    if(bKickMessageRecently && Title != "Kicked") {
        ConsoleCommand("Disconnect");
        `log(`location@`showvar(MessageType)@`showvar(Message)@`showvar(Title));
        bKickMessageRecently = false;
        return;
    }

    if(Title == "Kicked") {
        class'UTPlayerController'.static.SetFrontEndErrorMessage(Title, Message);
        bKickMessageRecently = true;
    }
    else if (MessageType == PMT_ConnectionFailure // PMT_SocketFailure?
        && FailoverURL != "") {
        // Try to failover
        if (Outer.GamePlayers[0].Actor != None) {
            `log("Engaging failover to URL:"@FailoverURL$"!");
            Outer.GamePlayers[0].Actor.ClientTravel(FailoverURL, TRAVEL_Absolute);
        }
    
        // Clear failover
        FailoverURL = "";
        return;
    }

    // Return to main menu when it's a "real" DC
    if (Message == "Your connection to the host timed out") {
        `log(`location@`showvar(MessageType)@`showvar(Message)@`showvar(Title));
        class'UTPlayerController'.static.SetFrontEndErrorMessage(Title, Message$".");
        ConsoleCommand("Disconnect");
        return;
    }

    WI = class'Engine'.static.GetCurrentWorldInfo();

    if (FirstConnError_TimeSeconds == 0 && !bKickMessageRecently) {
        FirstConnError_TimeSeconds = WI.TimeSeconds;
        return;
    }

    // SOUND BUG FIX
    // How long does the slowest server need to switch maps?
    // For now, we set this to plenty of time for testing purposes. (24 seconds)
    // IF this experiment proves to be a fix, then don't forget to tweak this value down towards about 15-21 seconds.
   // Until complaints return and then you backpedal for a stable margin, which is preferably also a little bit future-proof.
    if (WI.TimeSeconds - FirstConnError_TimeSeconds < 24 && !bKickMessageRecently) {
        return;
    }

    `log(`location@`showvar(MessageType)@`showvar(Message)@`showvar(Title));

    if (WI.Game != None)
    {
        // Mark the server as having a problem
        WI.Game.bHasNetworkError = true;
    }

    if (UTPlayerController(Outer.GamePlayers[0].Actor) != None && Title != "Kicked" && !bKickMessageRecently)
    {
        // Reconnecting as part of the sound-bug fix
        ConsoleCommand("Reconnect");        
//        UTPlayerController(Outer.GamePlayers[0].Actor).QuitToMainMenu();
    }
    else
    {
        class'UTPlayerController'.static.SetFrontEndErrorMessage(Title, Message);

        // Start quitting to the main menu
        `RxGameObject.LANBroadcast.Close();

        // stop any movies currently playing before we quit out
        class'Engine'.static.StopMovie(true);

        // Call disconnect to force us back to the menu level
        ConsoleCommand("Disconnect");
    }

    FirstConnError_TimeSeconds = 0;
}

So for now, the code changes that I've presented here offer a true workaround on my reproduction setup (without Vista fix) and is imo very much worth a try. The Vista fix oddly enough also works for me, while not for some others. So, fingers crossed that that doesn't mean that it's a different bug. Yep, that's scary.

Direct result:
No more sound bug. Map switching works great.
Nothing critical changed as far as I can see from here.
Triple-checked mapchange, restart, kicked, true DC.

Side effects:
It does break the recently added Failover functionality, which would ordinarily respond to such a connection loss.
But! If you scroll down this thread, you'll find that I've later added an Rcon command and properly integrated this fix with the FO.

These changes have been thoroughly SDK-tested with Dedicated + PC-client + Laptop-client.
I cannot guarantee the working of the fix, but it should at least NOT break the game after rollout.

This workaround comes from an even more informed position than the other two attempts, and has a decent chance at resolving the issue.

...

... and if not we'll just write a new engine.  (sarc)

Edited August 10, 2020 by DugeHick

[Suspiria]

"Final" adjustments complete.
More eyeballs please.
This one may actually do it.  

@Agent @StoneD

Edited July 27, 2020 by DugeHick

[Agent]

I'm trying to remember what our previous analysis was, but iirc this bug was narrowed down to how UDK makes a specific Windows API call during reinitialization of the sound engine. I think we were originally planning on just reinitializing it in a DllBind, potentially in such a way that would also allow you to select the audio device you want to use, as a bonus. I would have to reach out to the person who did that analysis previously to get finer details, since I can't seem to find notes of it anywhere anymore sadly.

Glancing at the above, it's worth noting that the failover stuff still goes through ServerTravel. It sounds like the idea is mostly to try to ensure that the client attempts reconnect after the server has fully loaded the level, in order to minimize the opportunity for the garbage collector to run? Am I understanding this approach correctly?

[Suspiria]

16 hours ago, Agent said:

I'm trying to remember what our previous analysis was, but iirc this bug was narrowed down to how UDK makes a specific Windows API call during reinitialization of the sound engine.

Well, there are no XAudio2-related calls between Browse and LoadMap.
And on my repro the sound bug occurs after Browse but before LoadMap.
So only if it turns out that my repro bug is somehow a different bug than the sound bug, would it be wise to uphold such a theory.

Note that when the voting interface displays "Loading Map", the client isn't actually loading the next map yet.
After 15-20 seconds it receives an invitation from the server and only then will it actually call LoadMap.
That's probably why everyone is thinking that this bug occurs while loading the next map.

This "narrowed down" tale that you're referring to, may very well be based on conjecture.

16 hours ago, Agent said:

I think we were originally planning on just reinitializing it in a DllBind

Even if such a WinAPI reinit function could be called in relation to the UDK process, most probably one would still need to finish the job within the engine by native-calling its audio init function (to e.g. obtain the new IXAudio2 interface pointer) which wouldn't be possible from script nor DllBind. As such I'm not so sure in what sense there would be anything within the order of "just" about this. If it were that run-of-the-mill, it would've been done back in 2017.

Unless we build our own audio subsystem, I'd say that if audio goes out, it stays out.

16 hours ago, Agent said:

Glancing at the above, it's worth noting that the failover stuff still goes through ServerTravel. It sounds like the idea is mostly to try to ensure that the client attempts reconnect after the server has fully loaded the level, in order to minimize the opportunity for the garbage collector to run? Am I understanding this approach correctly?

Almost.

With the caveat that the GC will not be stopped from sweeping.
Rather it will still sweep yet no longer prematurely sweep current-map related stuff.

 For some reason, audio-critical stuff gets marked stale once one calls (Pre)ClientTravel -> .. -> Browse.
But Browse isn't LoadMap yet: it will first wait for the server to accept the new connection.
And within that waiting period is where GC get's all opportunistic about our audio's. 

[Suspiria]

.

Edited February 11, 2022 by Suspiria

[Suspiria]

Found something!

I'm a little more confident about this one as I'm starting to better understand what is happening.
Also thanks to @Agent for mentioning the allocator a couple of years ago.

So, here goes the incredibly complex and arduous fix:

Add this within ProcessClientTravel within GameInfo.uc, (or override in Rx_Game)
at the beginning of the PlayerController foreach:

        P.ClientForceGarbageCollection();

Done.

Let's hope this fixes it before Firestorm. 

If this doesn't work we could still look into byte-patching towards some code cave from which to reinitialize XAudio2.
But hopefully that won't even be necessary.

Spoiler

How it works:

While the client awaits the server to switch maps, it suffers a window of vulnerability where its periodic GC sweep could crash XAudio2. By forcing this periodic GC sweep to occur just before travelling, this timer gets reset and the next sweep will come much later.

This also explains why the sound bug occurs so randomly.
If these conditions occur together:
- server is switching maps (like 8-15 seconds)
- client was notified of the travel with native ClientTravel (-> here starts the vulnerability)
- map duration including mapvoting was X min and 30secs or X min and 0secs

.. in goes the sweep and adios goes your audios.

So, in theory, this little trick here will allow the server a whopping 30 seconds to switch maps without the sound bug occurring on its clients.

 

Edited February 11, 2022 by Suspiria

[roweboat]

@Handepsilon @NodSaibot

[NodSaibot]

4 hours ago, Suspiria said:

Found something!

I'm a little more confident about this one as I'm starting to better understand what is happening.
Also thanks to @Agent for mentioning the allocator a couple of years ago.

So, here goes the incredibly complex and arduous fix:

Add this within ProcessClientTravel within GameInfo.uc, (or override in Rx_Game)
at the beginning of the PlayerController foreach:

        P.ClientForceGarbageCollection();

Done.

Let's hope this fixes it before Firestorm. 

If this doesn't work we could still look into byte-patching towards some code cave from which to reinitialize XAudio2.
But hopefully that won't even be necessary.

  Reveal hidden contents

How it works:

While the client awaits the server to switch maps, it suffers a window of vulnerability where its periodic GC sweep could crash XAudio2. By forcing this periodic GC sweep to occur just before travelling, this timer gets reset and the next sweep will come much later.

This also explains why the sound bug occurs so randomly.
If these conditions occur together:
- server is switching maps (like 8-15 seconds)
- client was notified of the travel with native ClientTravel (-> here starts the vulnerability)
- map duration including mapvoting was X min and 30secs or X min and 0secs

.. in goes the sweep and adios goes your audios.

So, in theory, this little trick here will allow the server a whopping 30 seconds to switch maps without the sound bug occurring on its clients.

 

I'll take a look at it. We had some interesting progress regarding XAudio stuff as well.

 

https://github.com/TotemArts/RenegadeX-Extensions

[Suspiria]

2 hours ago, NodSaibot said:

I'll take a look at it. We had some interesting progress regarding XAudio stuff as well.

 

https://github.com/TotemArts/RenegadeX-Extensions

Ooh, fancy!
That looks really good, Sarah.

I had located dinput8.dll through IDA before aswell, it had only 1 call indeed.
Perfect candidate to prevent triggering AV with injection, yes.

Sounds like a great plan B.

More technical talk about how to potentially salvage your library in here:

Spoiler

And if not for the sound bug (if it can be fixed my way) then perhaps can still use such a library to byte-patch some expensive Actor loops towards a threadpool that is initialized in DLLmain.
See if we can pump the game thread a bit lol. Hand off some stuff.

Loops that change data would require some mutexes and joins on the threadpool, but on those AllActors read-only loops it should easily work without all that. A couple of weeks ago I (re)started some profiling of the game thread but haven't specifically located expensive code sections yet.

But then again I'm not so sure whether or not it would be more rewarding to "just" *ahum* migrate the codebase to UE5. I've found that porting code can be relatively quick because you don't have to think of game design all the time. (although one probably will, still)

Also, another idea I had: to see if we can recreate the entire UE3 scripting system into UE5. (multithreaded ofcourse)
Yet I'm not sure of the legal implications of this.
Afaik it is really not allowed to modify the code in this way and then distribute such an SDK.

Edited February 11, 2022 by Suspiria