#564: EVE Online Departs for Python 3

0:00Every ship and eve online eventually undocks and leaves the station. This time it's the whole game. Eve is run on Python 2 since its launch in 2003. All 2.4 million lines of it on a custom stackless interpreter that stopped at python3.8. And stackless python was mothballed last year. Now it's destination python3 for eve online. The route runs through 6,500 lines of division that decide who wins a fight and 100 gigabytes of pickled python objects that have to survive the jump intact. Christian Cigarbergson was on the show 10 years ago. And he's back this time with Jamie Bannister who is flying the eve online migration right now and Thomas Dolling who took eve frontier through the jump first. In eve a destroyed ship is gone for good. There are no do overs and the universe has to stay online the entire time. That's a serious migration. This is talk python to me episode 564 recorded Thursday September 10th, 2026.

1:02Welcome to talk python me the number one python podcast for developers and data scientists. This is your host Michael Kennedy. I'm a psf fellow who's been coding for over 25 years. Let's connect on social media you'll find me and talk python on mastodon blue sky and x the social links are all in your show notes. You can find over 10 years of past episodes at talk python.fm and if you want to be part of the show you can join our recording live streams. That's right we live stream the raw uncut version of each episode on youtube just visit talkpathon.fm slash youtube to see the schedule of upcoming events be sure to subscribe there and press the bell so you'll get notified anytime we're recording. This episode is sponsored by centuries sear. If you're tired of debugging in the dark give sear or try. There are plenty of AI tools that help

2:06you write code but centuries sear is built to help you fix it when it breaks visit talk python.fm slash century and use the code talk python 26 all one word no spaces for a hundred dollars in century credits. And it's also brought to you by talk python courses course completion certificates are now live if you finished a course there's a certificate waiting for you on your account page right now download it as a pdf or add it to your linkedin profile with one click under licenses and certifications same section as your formal degrees visit training.talkpathon.fm slash account to see what you've already earned. Christian Jamie and Thomas welcome to talk python Christian welcome back Jamie Thomas hello welcome to the show. Hello hello Michael. Hello great to be back. Yeah yeah great to have you all here. So I'm really excited to be talking about eve online again. When was it? I think it was 2016 17 it was a while ago. It's been a few days.

3:082016 and I think we did even brought brought up the discussion point off the python tree upgrade. So it's been on the table for a while. Well we're here to talk about upgrading eve online to python three. Well version are you going to go are you going to start at 30 or are you going to work your way through all of them or are you going to do a little bit of a skip ahead. So we are going to python 312. That's what we updated to like the first step in between to stackless python 3.8 but you know when we did the work 3.8 was already end of life so to speak and stackless python didn't go any further so we decided to go to 312 and have no work on our infrastructure pipelines a little bit that any upcoming updates are a lot easier. So we have plans to go to 315 later to see for example. Oh fantastic that's really neat. I'm be excited to hear about that. Probably some cool performance benefits there and profiling and other types of benefits that you can get from those last few versions there. We're going to dive into why stackless in all that there's a

4:12there's a lot of history here and it's a really really big project. Before we get to the project itself and the upgrade and so on let's just start with a quick introduction for each of you. I'll let you do that for yourself. Christian you go first we'll go around the the squiers on the screen here. Okay my name is christian say your person I'm a director of gameplay engineering for e frontier which is kind of like shares the same technology stack as e1line. I previously been everything from a game designer to software engineer to technical director of e1line and later technical director of platform or share technology segment as well. So I was part of the team that did the migration for 2.3 for frontier and now I'm the primary benefit from it. So I'm Jamie I've been an engineer on e1line for about 16 years. Most long gameplay kind of that has led me work across almost all the different aspects of the game from the

5:16front end of the cloud almost through to the back. So I've kind of seen all the very good son of the glory inside e1line. So I'm at a moment working on the actual migration project for 3.1line itself building on kind of what what the others have done previously. Excellent promise. Yeah I'm Thomas Daling. I'm a principal programmer on the platform segment for fennelous creations and I have been leading the initial python 3 transition for carbon like getting us all to 7 on to python 3. Amazing. What a big project. What a big project. I think where we should start is just what is e1line and I have of course e1line but tell us what is e1line. Who ever wants to take this one. Perfect. So e1line it's an MMO RPG in the classic sense has been going since 2003. So it's a massively multiplayer online game set in the in a far future of where humanity has

6:18gone through the wormhole to a new system and society has kind of collapsed and then be built in a new form. It's a very what we would call in gaming an open world sandbox. So we support all sorts of gameplay professions from mining, trading, piracy, a lot of combat. I think the distinctive factor is the game is full kind of full loss. If you kill someone's ship that's a ship removed from the game. There's no kind of getting things back but nothing lasts forever. So the game is built on creation and destruction. A big part of it if I understand it right is kind of the economy and just the society. Like there's there's a society that forms from the players. Yeah the economy I think and before I joined to work on the game I was a big player as well and the economy was a big part of it for me. It's it's been used by or been studied by economists because it's

7:18if you think the data that we have on a big real economy on our economy is more accurate and detailed than you can kind of get from a studying real world economy is where you've always got imprecise data. So we have every single transaction we know everything that's going on all you see all the money as it moves around the game. That's super interesting. Yeah very exciting. What if we lived in a society where there was zero privacy and 100% data sharing. Yeah it would be creepy from a human perspective right that just like governments and I'll tell territories and all that kind of stuff but from a economics perspective you have perfect data right. Yes yeah and it's interesting how real world things have come up in the game but there's no formal mechanic for banks for example but players themselves have built banks to build kind of loan schemes. Organisations have got their own way of paying the members but turning up to operations so they have all the

8:20equipment of like we have welfare groups we have very centralized democratic groups, those communist groups all those kind of different social phenomena have reappeared in the game kind of organically. So interesting and many online games are you go here you find a server you join 40 people and you go have an experience right but this is a single unified one world that everybody is in they can all interact all that kind of stuff right. Yeah so we take place we call it a single chart so yeah like you say it's one universe any two people in the game if we go to the same place they will meet and see each other and so we have I think our log just kind of play a group is about 15,000 characters and so all of those potentially could get together and go to one place at one time and have a huge fight. I think we have two Guinness World Records for the most people taking part in a PVP fight. Wow how do you coordinate that as a group you know what I

9:26mean like how do you coordinate just the conversations amongst the people without it just being a cacophony between and turn ourselves I mean with the players. I mean with the players if there's 1500 people in like some sort of action how can you have conversations and communication without it being built their own solutions for a lot of this so they've replicated a lot of a chain of command that you get from from wheel kind of real world and some of the bigger groups they have their own IT departments but take care of things a lot of groups use discord to kind of coordination but they have and various other kind of voice systems that they have what's called an FC which is like a fleet commander so that's the person giving the orders and kind of directing the troops and then it kind of there's a hierarchy if you like it communication it goes down so yeah the bit the bigger groups have built very complex kind of command and control systems themselves.

10:29I guess that makes sense but wow and you think of a real army if there's a thousand people they don't all just get onto with the same audio channel they do have certain people talk to each other then they give commands down the line and interesting. Yeah they replicate the process of like scouts filtering information up they'll have some people gathered information and feed it into the FC who that makes the calls of where they're going to go what they kind of do next. Yeah Christian Thomas you guys want to add anything to that overview before we dive. Yeah I would say that like this doesn't really happen by its own like there are like a lot of people that play even online that spend like incredible amount of time kind of orchestrating all this not only just the fights themselves but the the research behind the fight gathering everybody up so it is really fascinating to see and to some extent these people are playing the game but they're not actually in the game because they are managing the spreadsheets and the communications and all of that and to some

11:31extent in certain circles like people have put their experience in carmanaging these larger corporations in the end on to the CV and it has helped them get a job like that's the level of commitment there. Yeah I can now function as a full logistics operation person because I've done it for a thousand people in e1 line. Thank you so much. Crazy Thomas. Oh I've got nothing more to add there I think they both discovered it really well like it can almost be almost more like a job than a game if you want it to be. Yeah you know not a lot of valuable things you meet a lot of great people that way as well right I mean over the 20 years so many people that met each other like we even had like a wedding at one of our fan fest I believe so yeah we've actually had the few I think. Thomas you gave a great talk at fan fest about upgrading and sort of inspiring the folks and that video is online in fact I've I'll link to it in the show notes we'll maybe get

12:32to that in a little bit later so yeah that sounds like a really cool gathering you know I really last couple years I've really enjoyed just diving into games and my daughter is getting older and what are the things she really likes to do is sit down and play games together not even online is it's too too big but you know sort of cooperative stuff and sort of exploring and I think that there's well I know that studies have shown that there's more loneliness there's fewer third places that people find themselves at and all that having something like this where you can gather and be with people and sort of immerse yourself in I think it's really neat honestly always you don't take it too far you know it's still about a work stuff yeah yeah I know before I joined an as a player we would organize meetup events at different places around UK where I was from and so I got to know a lot of people I traveled to a lot of places just for the purpose of meeting up with other e-plays so for for a lot of people it's it leave or or you know a lot of games like this they can be your primary

13:36social connection to it so that let me take a let me take a smack 35 years or something like that have to the days of bbs's you got if any guys don't bbs's remember them I yeah used to play the muds back in the day oh I used to play muds as well though that's not what I'm thinking but actually muds have a really interesting multi-user dungeons they have some really interesting analogies as well but I was thinking of trade wars to as any of any of you all played trade war from the bbs days I know of it I've never played it it's it's really interesting it was this multiplayer thing way back in the 90s and you would dial in and you'd have to take turns because only one phone could be connected to the bbs at the time at the time at a time and it was really it has some real similar vibes it's you know you travel to different planets so there's mining there's an economy there's trade anyway I feel like this is like what if it could be real instead of just a little sort of simple game what if you could make it like real with all the nuances

14:37and intricacies of reality and multiplayer and let's just it's the the realization of what that thing is for the whole thing. Yeah and I was you you could conceive it was heavily inspired by a game called Elite which was one of the very early 3D space games and that that had a big world of a big universe of systems trading was a big part of it and so there's definitely links from that and also classic role-playing games as well our character stats have similar kind of perception intelligence charisma sort of stats. Interesting almost a D&D sort of very much yeah yeah you can tell it with D&D players in the very early you've designed us for sure. This portion of talk path into me is brought to you by century and seer AI there are plenty of AI tools that help you write code but century seer is built to help you fix it when it breaks the difference is context seer isn't just guessing based on syntax it's analyzing your actual century data your stack traces logs and failure patterns because it has the full context it can

15:40a spot buggy code in review and help prevent issues before they happen and b identify the root cause of production errors it can even draft a fix and hand the work off to an agent like cursor to open a PR for you seer turns century into a complete loop you have your traces errors logs and replays to see the problem and now AI to help solve it join millions of devs at companies like quad Disney plus and even talk by thon who use century to move faster check them out at talkbython.fm slash century and use code talk by thon 26 all one word for $100 in century credits thank you to century for supporting talk by thon we're going to also talk about this thing called eve front here which is a little different but very much inspired shares a lot of the code base so give us a quick comparison to eve frontier which i'm sure fewer people know about uh yeah sure so one of the what matters for the tech is that we've uh essentially for even line the code base and are

16:41siering a lot of like the underlying tech we've made a lot of modifications we haven't been kind of pulling in stuff uh lately uh because it's just it's become too different where it differs from from even lines maybe if you if you play even line you very much come into a society part of it is but definitely play a best but part of it is also kind of uh and be seep it the environment is there's a police there's everything whereas in front there you're more coming into like an unbased uninhabited zone and you're the first kind of writers in that universe and it's up to the players to build that society up and and kind of build up that all of that infrastructure uh it doesn't really differ from even a lot of like the uh the design part and i think both games aim to give players as much player agency as they possibly can but it's also allows us to make like a lot a lot more dramatic changes in in the in the gameplay itself it's very hard to change the gameplay on on even line for

17:45example very drastically because honestly like after 20 years that's the game that people are playing they don't really like they didn't play it for 20 years to change all of it uh so uh we've been doing that it's more survival survival based it's uh more like uh like a survival game than it is uh uh uh not necessarily more than an MMO but it's definitely has a lot more survival elements to it yeah and the environments are a little bit more kind of claustrophobic and more obviously that's very hard in a space game because space is pretty vast yeah amazing okay really really neat um now let's talk about the reason that i reached out to you which is i saw this article or post from you all that said the move to python 3 begins and you know credit to you all this is a really fun article that it's like it's it's told as sort of uh the next phase of a game or like something where it's kind of a mission in the game i guess but a mission for you all right yeah i think that and we kind of got a little bit uh cheeky with the heading because it isn't

18:50necessarily the beginning because uh like we say now that uh frontier itself is already on python 3 but this is specifically for even line itself uh is now actively moving to python 3 and a lot of is building upon the tech that we've developed already and kind of start to prove out through fun too uh but yeah this is kind of our our big formal announcement that the legacy of 20 plus years of code we now kind of actively working to demigrate i think we should start maybe with a little bit of the architecture and the just a big tech picture of eeve online when we spoke christian way back in the day you were doing a bunch of interesting things i think you must have been the lot single largest deployment of python a python app on windows is that correct i would say yes right well what else would it be but that seems very plausible i think uh like it was a very kind of like a niche environment and what i'd often run into is that i would try to take some libraries and try

19:52to use them and they just wouldn't work conventors like they somebody broke it and nobody cared because they were really using it on on windows except we were uh obviously like at that time like 10 years ago we couldn't we couldn't really use a lot of like especially native python libraries i can't even remember if we had removed uh we must have removed we had our own importate as well which made it also difficult to use third party libraries so but i think we like fixed that by that yeah there's a lot of i mean it's a very old project and a lot of the early beginnings in the code base were before a lot of the things in python actually been standardized to a degree plus the special circumstances of of having a packaged program with its own python interpreter just makes certain things more interesting like you know as christine alluded to like you can't just import a random public package because we don't have pip as an example right or you via these days but like there's a lot of a lot of interesting problems you need to solve when you have such a large

20:52for sure yeah there was a lot of custom stuff when did this first get created so i know we spoke in 2016 but it's it's been around for more like 20 years right so 2005 2006 but i think it's earlier than that it's like the game is released that 2003 the development started in 99 i think uh they didn't really choose python i don't know the exact date of that but like uh it was uh that wasn't that there's an interesting story as well like why they chose python of which was actually rather new at the time yeah i was actually wondering about that as well i'm glad you chose python i think it's really cool and i'm sure you all really enjoy working on it and this is this has got to be an exciting project to say we're moving to three fifteen we can use uv we can uh import libraries like all the exciting stuff but at the time it was pretty new and they said well we could run this much faster and easier if we wrote it i don't know with c++ java csharp something like that right

21:57i think a lot of the um i kind of call libraries uh they are an in c c++ so physics engine uh which we call destiny that's that's all uh c++ uh trinity are uh graphics engine uh similarly um but yeah most of our uh kind of higher level uh all the game paid logic um a lot of the network stack but we have um that's so billar or clustering stack that's so billar in python yeah probably add to that as well is that like very the time when we started or not me uh even though i've been here for a very long time it hasn't been that long but when we when they started writing it like that was around the dot com era and kind of the older listeners probably remember that era but uh programming resources were scarce and heavily sought after so uh very expensive so part of the reason why they chose python was to get more junior people and even even designers to be able to write

22:58sequential code in python in order to like speed up development it was all about development especially in Iceland right where you've got a much smaller pool uh of people to work from honestly that's that's kind of a cool success story right like bring more people in to help building build the thing that we're building yeah absolutely absolutely and i mean as as christian just said like you know allowing people to write sequential simple straightforward python code as well which then uh led to us uh while discovering stackless python integrating that one and then building an abstraction on top that people don't realize that they are writing code that runs asynchronously um and you know it enabled us to scale to the to the degree that we were able to scale right with thousands of people like being in the same solar system and so on yeah yeah you you needed concurrency before python was inspired by concurrency and maybe we we should talk a little bit about free threaded python later as well as the possibilities there but i just want to

23:59kind of touch on this historical angle a little bit more to me there's two really significant milestones that matured python i'm sure there are more but these are the ones that stand out in my mind they both are around 2006 or so time frame so one is jango coming to python really making it a a first class way to build web apps people start building on top of that that puts a lot of pressure you know make python operate a little bit differently right we 10 years later we've got instagram running the largest jango deployment in the world and doing things like trying to juggle weird memory stuff by turning off the gc for example entirely and other things along those lines and the other one around 2006 as well as the invention of numpy and then later jupiter in the whole data science stack which also put really hard computational pressure on python rather than say the parallelism of web apps and so on you all were before both of us right so it was early early days yeah i mean what we what we also found was or what i found in some of work

25:06teams the original developers had to invent a lot of things which didn't exist so like the containerized system a way to kind of cluster an application across multiple CPUs and notes there wasn't really a lot of easy official framework for that kind of thing so they had to roll their own even logging and telemetry systems there weren't great ones of those so what we found from the early years is we have a lot of in-house tools which we made or the people of the time made which are now though using the sort of the models that you see have been kind of replicated by much more popular systems so whether it be redis for a kind of storage system or for your containers as well we're aware that we remember we can talk about this in a bit more on a bit is the way we distribute eaves as an application across about 200 notes in our server cluster and each of those has

26:09certain jobs to do and so we have to deploy and configure all of those as one giant cluster we have our own coordination tools that have been built out of the years nowadays if you were starting this from scratch you would have so many official solutions that we just didn't have the option for. Oh you just set up Kubernetes and then you do this auto scaling thing and it's oh yeah it's just easy yeah are you also sorry I feel like the data scientist kind of grabbing onto Python and I feel like pushing it towards Python 3 because I think like without like that army car plug going into I feel like it mirrors the rational of why we chose to go into StarClaz Python. We were giving basically a tool to a bunch of people that aren't really programmers but they need to like write code so I feel like that that is an interesting thing. Yeah you know I've forgotten the 2v3 and all that now yeah it's easy to forget the details of that from it's been a while now actually

27:12but you're right it was the data scientists that were going into Python 3 and the rest of the folks were kind of dragging their feet because they had existing code bases that needed to keep running and keep being migrated whereas the data science projects are more frequently like we'll start a new project for this question to answer this question or to study this new bit of data so they they had much many more opportunities to choose the most modern tools and I think honestly we should give the data scientists a little bit more credit of making the 2 to 3 change happen yeah. All right well let's talk stackless Python because this is one of the the foundations that you all had to migrate from and you're the the the journey begins the move to Python 3 begins article talks upgrading from 2.5 to 2.7 not too long ago and then finally through up to 3.8 what what is it about stackless Python I know it has strong origins with you all because some of the core developers

28:12worked on it and refined it there but this far down the line I imagine there's a lot of people that don't know what this is yeah absolutely stackless Python itself was a very great solution at a time right I mean it's basically go-lunks go routines but in Python before things like the this co-routine model became so popular as it is in recent years a big problem that we faced was on the one hand that because of the customizations we had to do to the network stack to make all the magic of people write sequential code but it does end up not blocking the main process right that was a big learning curve for a lot of people and also always something that even the experience developers tended to like forget about in a moment and like oh yeah hang on there's a DB called of course that we healed I need to put a lock here or whatever right so the model in itself is great but it comes real for with a few like learning hurdles I want to say and it served us

29:16really well one problem that we had over the years with that had become more and more interesting especially in the last 15 years I want to say is that we relied on a heavily customized version of it so stackless Python itself is a fork of the CPython interpreter with a lot of changes in the interpreter and we put some changes on top because you know doing a real time game of sorts requires some solutions that go very far down the tech stack to be performant like if you want to get certain metrics if you want to control how memory is allocated and so on and so forth so all these customizations and also meant that made it very difficult for us to like always go to the next newer version of Python as the article covering codes out like you know there was some code that was still Python 2.5 when we did the native extension porting to Python 3 we discovered things from Python 2.3 that we were still using that shouldn't have been used but we somehow kept the wrong thing or customized version of of stackless Python. This portion of talk Python is brought to you by talk Python courses here's the thing that always bug me you finished one of our

30:20courses that's hours of video a pile of code you actually wrote in real skills you didn't have a month before and then nothing happens no paper no credential nothing to show for it so we fixed it every talk Python course now generates a completion certificate automatically go to your account page in your dashboard section scroll down to your completed courses and click certificate that's the whole process two things you can do with these course completion certificates download the fold pdf which is handy if your employer reimburses training or gives you credit for finishing it or you can make the certificate public and hit share on LinkedIn which adds it to your LinkedIn profile under licenses and certifications not a post that scrolls away in a day an actual credential sitting on your profile where your manager and cruders can see it plus if you've been taking our courses for a while you probably earned several of these without even knowing they existed just visit training.talkbithon.fm slash account and collect them thanks to all of you who have taken a talk Python course it's a great

31:21way to support the podcast another aspect certainly is that stackless had become like a sort of niche project in the ecosystem overall right but Python 3 we have async IO built on the on the twisted model and even in the Python tool world a g event had become a lot more famous than stackless Python presumably because you could just import g event in a standard c python interpreter while stackless Python is more complicated to get started with so stackless python was basically its own runtime yeah it's basically yeah yeah very much that it's like its own Python runtime you kind of need your own like it has a stackless lip so to speak which basically monkey patches similar to what g event does all of the threading and then socket modules and so on to be able to do them in non-blocking fashion and yeah all these things so that's really that's what made it possible that's really awesome one thing just I don't know if you all know this but how is it possible that it says it's forked from

32:23c python on github when c python itself didn't go to github until like the 20s you know 2020 I don't know what it was but maybe some github magic I don't know how it's git magic you will find on github they are repositories we are they are commits from abra and link colon and various other historical figures because you can just make up all the history right interesting yeah that's it is nice though actually that it shows it if you go to the repo now it says that it was archived in 20 25 which you know not every not every project is meant to last forever but it made it possible for you although I'm sure that you would you would think you know right you're like wouldn't it be cool if we could you know use I don't know libraries like hydantic or fast API or aioreta so I don't know whatever right anything that is new and fancy and then you go back and you look and you're like well but not here not on sack list right like that was one of the major problems wasn't it yes this was a major headache for us but for every native extent or like

33:26not just native extensions for every python package we took on we had to like take a really close look of about like how can we integrate it in our stack like again like we didn't have a package manager we still don't have it but we're working on it and they just meant that you would take the zip file from pi pi for example put it no extract it into our environment and then see what happens if that then happened to be something that had a native extension well good luck does this support our minimum spec environment does this one on all our environments right does this work with our version of the forked of the forked python interpreter and yeah we have we have discovered a few few cases where we took on modules that were built for two seven eighteen and they then corrupted memory because the in two seven six the memory layout of the pi type object changed and we didn't have that change and yeah things like that so we had to fight the vote forward. Jamie you want to jump in? Yeah now I'm just going to say when it comes to support as well because we've we support Mac and we have in the past of where we don't now support Linux as well so

34:30sometimes with a native extension we've got to figure out does it even support all of our clients our service was done on a Windows cluster most of our players are running on Windows clients map is also is kind of a smaller majority and I can say we used to also support Linux so keeping all of those in sync as well with with all of these extensions has been probably that a good times yeah sure that's an understatement earlier when you guys mentioned how you deploy as an app or you know dot app value xe whatever or or to your server clusters how do you do that that's honestly one of the biggest challenges a python that many attempts have been made but I honestly don't think that there's a good answer has a really foolproof answer yet and until it's built into c python itself with a like a dash dash build flag or something I think it's it's going to be 90 95 percent solutions but you'll have something that works what are you doing?

35:31So effectively I mean with pifin3 this all got a lot easier because you can initialize the interpreter a lot more straightforward but what we have been doing for the last 20 years is effectively we are embedding the python interpreter in a custom binary that we have custom library that we use and from that moment onwards we basically just heavily customize what paths are available to to the interpreter and you know what environment variables are considered all these all sort of things and then we package up the remaining things that we need in a way that it fits this layout that we set up in in the embedded interpreter but yeah it's also more like a 75 percent solution than than a real one because the python build process in itself requires some system specific paths like we had to go to great lengths to to patch that out not necessarily in the best way possible either but you know it gets us there if you're shipping it that's you know that's off you all that's pretty pretty impressive there was kind of like that we have a solution for our players but also like we built a lot of tools for like game designers to use and stuff like that

36:36and it's convenient to write them in python because we can share some of the code base of the actual game game logic for example when we're creating like static data and that has been like a very painful experience of distributing python like python basically a python script to develop us because there is really amazing how much people can mess with their environment by just changing the python path running the wrong version of python it's it was like it's been a struggle for years honestly even if we get a new hire the first few days are often get spent setting the machines up and inevitably as windows slightly changes from you know released to release it will or won't come with certain things like we sometimes we have to disable the built in python that ships of windows just so that we can use our own specific python to seven there's a windows three python three so that's also quite a lot of fun we have to have to go can we get sort of out what about things

37:37like UV run or UVX in the future you know but there's some interesting possibilities there potentially absolutely and as I said we are working on it we're exploring the the angle there because it is still a bit of a hassle of getting things into our our special environment for the game itself we have historically had also the issue that we were using a different built environment than the standard python distributions right partially because like there's a lot of code that we need to be migrated partially because it works we can't we don't want to spend resources on that at the moment partially because well you know especially with stackers python you could only go so far until there's a new compiler version that then breaks some internal and you would have to do major re-rides so these issues have existed but we are we want to be able to be in a place where a developer can also just say UV install this package into my game environment then be able to use it

38:37needs to solve a few things like if it's an native extension you know it needs to find the right built environment and and so on which is non-trivial unfortunately but what workflows do you think do you see becoming possible that were much much harder once you have nice package managers and maybe more reuse right for example you could ship versions of things more easily than you know as one monolithic element yeah for sure um I mean for example we recently open sourced basically the game engine right and uh there is I can see a world where you can just UV install or pip install these things into your custom interpreter to get going by into your python interpreter to get going right without requiring our custom interpreter um for our internal workflows I don't think I mean sure the the acquisition of an of an external package will definitely get a lot easier but uh many of the other workflows probably won't change that much since we uh I was in 2011 or 12 or so we added uh an interpreter mode to the game engine itself so you can basically start the

39:39game engine with a special flag and then it basically starts to pif and rappel and you know you get as much of the standard hyphen interpreter interpreter behavior as possible um yeah that's course you can kind of poke around that in the the game memory and values and data structures as you want right I guess the director and me doesn't really like uh it doesn't see it or it's all negative that there's a bit of a hassle to add the new dependency that's right you know I was thinking about that too Christian I was and you can see these headlines of all the supply chain issues and you're like well I mean we should worry about it but not to the degree that other people with 200 dependencies do right yeah definitely like if you make like and I'm not like I'm not arguing necessarily for making it hard but you can see like a reality when you have a large development team and people just kind of uh uh flihpantly at the dependency that causes issues very like it's very vulnerable to like individual mistakes yeah and also just duplication of now there's three ways to do three

40:42libraries that kind of handle the same thing but on the flip side if it's harder to upgrade you're also less likely to actually upgrade it better what libraries or or tools or experiences did you all see out there and you're like all right we have to upgrade were there certain packages you're like oh gosh we really really would benefit from using this one or having this feature like the jit or something in python that makes such a difference that it's worth all this effort so I mean Jamie maybe you can talk about the python side a little bit but like from from the game engine perspective the main driver for like really wanting to get the upgrade project pushed through was the native backline that we worked on in the early 2020s um it has just like python 27 was not a way of silicon right to begin with um then a lot of other changes that had happened uh like macOS behaves very different with unicode then windows does when it comes to system calls and so on

41:45historically before we had the native macOS clientus would go through like a translation layer similar to wine for linux right um so we really pushed for it because we just saw that okay if we don't go up to python three we are stuck with having for have like a a small team that is dedicated to nothing else but maintaining the python interpreter and uh that seemed like uh it's one possible solution but not necessarily the best one if you want to want to just go rapidly uh then add new features or iterate on other existing functionality um i mean for me as a as a gameplay programmer who's mostly using uh the python three mode at the behalf um things that we've solely been lacking for a long time is a good way of uh type declaration it's it's sort of there in python 27 if you're using the right IDE uh but it's not consistent i guess a lot better as you move through python three versions um another thing we end up making a lot of is uh what's effectively data class uh where

42:46we just want to kind of define a small packet of data in in a particular structure and pass that around do things um having the kind of the language support that uh a lot at a more fundamental level is something i'm really looking forward to getting yeah that's really and think about such a large code based was it 2.4 million lines of code something like that that's that's non trivial i mean that's kind of the scenario that typed python is perfect for and could have been invented for before AI was a thing which it also helps but having the ability to use tools like pyrofly or ty and ask the question in an automated way is our code still consistent that's pretty neat right yeah i think given the size of a code base and in the fact that um we have reasonable test coverage but we don't have complete by any means um so the more we can put in a kind of leverage from the language to help prove correctness and then find issues uh that's going to lead to a better product and less issues yeah last question before we actually get into the the details of the

43:47the migration and how you all did it what about free threaded python you started with this concept of a concrete high concurrence with so much so we need concurrency that we're going to get our own runtime and maintain it are you considering free threaded python is this interesting in any way or are it all a synchio driven what are considering here uh we are looking at it like i've actually just been to europe i from back in july um and i attended some of the the very interesting presentations on where free threaded is heading um there is a lot of work ahead of us to adopt it no because there are many places which just implicitly rely on the gill for synchronization we need to identify those there are some interesting parts where i think we could benefit from it when it comes to our resource loading system for example um but yeah it's it's on our radar we are interested it's not necessarily going to be the biggest like performance improvement that we could possibly get i think but um you know once it's there we also have more more possibility or

44:52more options to explore as well and then what we might be able to do different yeah i imagine that async iow given how much network redis etc etc stuff you do will give you a pretty mega boost as well as just the performance of c python has gotten so much better right yeah we've gotten like what was the 15 to 20 percent improvement across the board uh for most drinks like there's there's the one like string handling obviously changed you can't really compare that with python 2 because now it's all unicode that always bears a little bit of overhead compared to operating on on bytes directly um but so far we have been very happy with with the performance which will have gotten that'd be basically gotten for free right um on the top of for async iow though like unfortunately we can't really benefit from that because you need to annotate your functions with async and and so on and uh you know with the 2.4 million lines of python code written without that in mind uh it had become quite quickly quite clear that uh unfortunately that one is out of

45:55the window for us how interesting we did i we did uh discuss it at the time like whether it was it just that's uh completely out of scope of everything that we were doing strong you know one thing in terms of colored functions that is one thing i i just talked to Joe from the greenium project about tone iow which is um multi threaded async current time in python written in rust but implemented in rust but it has this way of running it without necessarily decorating the thread anyway maybe i don't know maybe it's interesting who knows give it a i intend to just talk at your reply from like oh yeah yeah okay they are and uh it looked interesting um but on the other hand we are doing a lot of similar things underneath the hood already like so basically all of the iow already happens on multiple worker threads in the background um yeah really just that we have i mean the main reason why the main way that we've has to let it stack less back then at the new schedule and what you'll know is that it acts as a synchronization point to get the data from those worker threads into the main

46:59thread that's running python and uh i've looked at tone iow a little bit but even that is like not very straightforward to integrate in our stack unfortunately so doing comparisons is a bit difficult yeah honestly and the free threaded stuff honestly it scares me uh not i i used to do tons of eight you know concurrent programming threads events signal you know locks all that kind of stuff it's not my code that scares me it's all the libraries that are written in python that like you all said a lot of people assume the girl was a thing and how many of those have some little latent race condition that when they figure out what it is they're gonna put locks on it then it'll become a latent lock uh deadlock you know what i mean like just having faith that all the foundational stuff has really been through the ringer of free threading i don't know it scares me i'm optimistic for it and i want it to exist but it scares me yeah there's a lot of work for sure um you know also like the other side that the single threaded use case shall not degrade in performance either you know it's there are many things but that being said like even with the asynchronous programming that we use

48:01underneath the hood to be are already in the world of having to be careful what we put a lock on and whatnot right uh yeah to you guys you all are in this world but i think 99% of the python people don't even think about locks and and and race conditions you know and that's that's i think that's where we're gonna have to as a community work through the issues to to make these dependencies stable you know yeah absolutely there's actually i think there is a community of what effort going on as far as remember uh where um is it is to you well really are we free threading yet or something um so there's a project where they basically track all of the public libraries in in in pypy and how they behave under free threaded python already so we'll be also keep our eye to keep an eye on that as well to see where that is having and then where there may be some of our dependencies in there there's the compatibility the free threading compatibility status checker it's got like the tested in c i the pypy release it's when it was first supported the free threading bit it's very data science

49:01heavy though isn't it yeah i guess it's once again the data scientists are leading the way here i guess they're also mostly benefiting from a free threaded python given that they are doing a lot of number crunching right exactly because i think a think i owe even though it may not apply to you all does solve the problem of i got to talk to the database a whole bunch of times concurrently or to a you know um microservices concurrently because that's the a i o bit but when you got to do computation it doesn't help you a bit so yeah you're right i think that's that's fundamentally why all right let's talk let's talk the actual migration so you all started by just making sure well first converting everything to run in two and three right tell us about that yeah all right so i'm going to test um so a lot of the the work that kind of came our way was to uh take our 2.4 million miles of code that we have and bring it forward so that we can eventually end up everything on python 3 what we found is like we mentioned early on we still have a lot of

50:05code that was technically 2.4 2.5 uh or using 2.4 and 2.5 kind of idioms um which are no longer suitable in python 3 so the very first step that we've taken is what we're calling legacy to two seven which is get all of those legacy bits of code up to two seven standard um because a lot of those changes then become more forwards compatible into three um that's we've actually uh already released and deployed the first part of that a few weeks ago that's around a time that we write up up and and for a lot of that we uh use these future ice groups um hold a lot of fixates from future ice and and what they do is they take one particular idiom of python uh and update it to python 3 there's a subset of those which you can run while still being two seven uh compatible um and so if those are the ones that we've done so far yeah and you just basically got them all all moved to a way that theoretically would run on python 3 because for people don't know

51:08some of the older ways of working in python 2 were literally incompatible like ways of to declare exceptions handling some of the string stuff um yeah we found a lot of uh was was testing membership in addiction week so now you do if x in y uh in the older python you would do if there was a has key function um that is no longer supported in in python 3 but we still have a lot of code or we did have a lot of code that was using has key so we had to change all of those to using the in-state in that for a time um so that's the kind of thing that we've to kind of done in this first uh this first phase um what we we found was we were actually 94% of our code was technically python 3 syntactically valid uh we've now got up to 99.8% so that's that syntactically valid it doesn't mean it's going to do the right thing when you put it in python the classic example is division um being a game that does a lot of things with a lot of numbers uh we have about

52:13six and a half thousand lines that feature a division uh some of those are going to break if we just switch to python 3 without doing anything with them uh because uh the way it changes from from integer to to float division uh so that's going to be part of the next next phase that we can get into yeah that was one of the main main deals if you have x slash y and python 2 that would do basically integer division truncated they take the floor of it more or less right and then those just become floating point numbers which yeah that's that's terrible for like this thing equal equal that thing and all it's and if you're doing a lot of like uh damage calculations you've got a one ship shooting another ship uh that can make a lot of difference if suddenly your damage changes and now I should have won this fight and now I've lost it because the math is working out differently so yeah that's uh gonna be important for us to get right yeah absolutely okay so what what what next what happened after that you got it working on python 2 but syntactically python 3 yeah okay so the next phase which we're now in at the moment um is uh we've forked

53:20the the python 2 crunch into a python 3 crunch uh immediately many many things failed all of our tests started showing up red and so on and so what we're calling that is our python 3 unstable and so from there we need to work through uh get all of the tests passing start getting that more stable at the same time we've got what we're calling a 27 unstable branch where we kind of introducing a lot of the the shims and the adapters so this is where we can have covered what's gonna maybe fork if to do this if three do this uh the downside of that is it's gonna be a lot slower so that's more of a compatibility proof versus a releaseable version um but that kind of is gonna be our middle ground for code that we can then say uh we can put this into 27 and make it force compatible or this is gonna be breaking compatible so we need to put it on the on the three side um and yeah that's our next however many months of work um but the other big angle of it which is really worth calling

54:22out is it's not just the runtime code we have a lot of data uh that is uh either data at rest or data going across a network that is serialized python um we need to get all of that compatible so if we have uh for example python to alter our classes and we want to serialize that to send into your python 3 note well that's not gonna work because python 3 is gonna what's this i don't know what to do with it so as well as changing the runtime code there's a lot of data at rest and data on the network that we are saying change right you did mention that you're seeing redis and one of the real common things to do is just to pickle the objects and put them into redis but you know what memory shape changes from version version i was literally writing a blog that's gonna come out in the next week or two a public blog uh talking about exactly that problem that we have here is that the agent blog? yeah so um in any of we have a bunch of NPCs um do i have to do a recall

55:23agents nothing to do with lulam agents do they just characters um and they have memory so when you talk to them uh it's gonna remember that you talk to them and then later on you're gonna come back and they'll give you a mission or to go you know go and fetch me this or go and kill this um it saves that bit of memory in the database and the way it saves it as it is as a serialized python blog though that when it comes back we can kind of uh really really load the memory or reload that bit of memory from the disk um we have those going back now for 20 years uh i think there's about 100 gigabytes of python memory in the db of agent memory in the db in the form of pickled python objects uh so that's that's a uh a big part we need to to go through and uh because we can't just write a database script that's gonna do those right because a sequel script isn't gonna be able to serialize and deserialize python objects it has no idea yeah yeah are you considering moving to something that's version independent like message spec or yeah um so somebody's or chase on

56:26so many things what we want to do is change the uh the format to uh what we're prototypuff because we use that for a lot of our uh gunner communications and and storage as well uh if a prototypuff or flapuff are depending on the size um but yeah a big part of our uh internal external APIs are a well built on top of a preva so we probably use that yeah so that'd be really natural maybe you could even just not even deserialize it just give me the stuff and shoot it back over the network if it wants in that format anyway digitally i guess all right we'll get really short of time maybe just take us take us through the remaining arc of of what you all planned out what what's what's still in the story to be told um so yeah so we've got the data formats that we need to change we've got the runtime code um once we've got those worked out uh some of the things we want to start doing are what we're calling mix mode which is where we take our cluster of 200 nodes uh plus all of our clients all of which are two seven at the moment and start to be able to selectively uh replace one

57:30piece of those with the Python 3 equivalent so rather than doing a big deployment day where we say like okay this is the moment uh we want to incrementally do it uh kind of spread that risk and upload it at the time uh but can you imagine uh the stress of just oh yeah i think the whole i've been there for as big deployments and uh very stressful uh so by doing this we can we can have to spread that at risk and that load over a lot of time but it also means we've we're developing a lot of tools to support this project that aren't necessarily uh absolutely required but they're going to give us a lot a lot better uh abilities so like the disability to deploy uh kind of a heterogeneous mix of types of uh of builds uh and even mixed do we want to do some nodes on Linux and some on Windows for example um by doing these tools that's going to give us a lot of these options yeah i give you more deployment options and so on right that's pretty interesting it's an interesting consequence i didn't really consider coming along but yeah it makes sense all right christian what else we want to say about this migration from i maybe from the the director

58:34yeah it's i mean i think it helps a lot that we did this for frontier first because one of the problems we had when we started this is that like we could start my migrating the python code but none of the python code would really work without the c++ modules being um um importable to python as well and we could start in the python those c++ modules but like they wouldn't really make sense without the python glue kind of trying it together so uh it was really convenient to do this on a project like frontier where there is risk of uh breaking things uh is uh is a is a lot less and like like the first release we did we had hourly downtime because after like one and a half two hours the the process with just some of the process some process somewhere with just stock responding at the all and we were like built in the dark there it was like uh it was very stressful time and it turned out to be something really really stupid so as it obviously would have but so i think uh i think the team city or the build system helped us a lot obviously being

59:41a game with you can imagine we don't have a robot robot suite of unit tests for all of our code days like i mean even automated tests wasn't really like a big when we started doing this and a lot of the cultures written in the first place or doing like an incremental thing was kind of hard but what we did do when we we started we started just running down the build system and just getting further and further into like an actual build so yeah yeah and thirded test graphics and physics and stuff through the automated tests as somebody could be right and when we started when i at least when i started i started talking to Thomas about this uh when i was a technical director and even and we were thinking about it for for even it's just the the entire concept of doing it just that's so just so daunting and like big better like couldn't imagine how we would do it we did have a good plan i think and that plan kind of uh we we more or less did that for frontier but obviously like the

1:00:45stakes are higher for human minds or like it's the difficult thing i think a big difference so or a critical thing for evis it's an online game so the servers need to be running every single day um we we kind of thought to be taking the server down for a few hours every day we need to kind of keep the product up so that's where frontier kind of running ahead of us was really really helpful because they could prove a lot of this out um before they got to the point where they needed to be online 24 so yeah it saves it saves uh it gives you a lot of flexibility to all that testing doesn't it yeah absolutely especially because you know the transition to pivants 3 is so i can all or nothing step you can't just put like one of your like you can't just put the base interpreter on pivants 3 and then keep all your native extensions to pivants 2 right you need to go all the way otherwise no it just doesn't work and then that that was actually the thing that gave i think there was like the the initial very biggest concern that we had like can we actually pull this off by by doing this push out all at once and i think it was also the main reason why we did this intermediate step

1:01:47of going to pivants 3 8 stackless pivants 3 8 first because then we knew okay we are only we are only changing the pivant work major version we're not also changing the async library underneath the hood which you know a lot of the code still has some timing dependency somewhere right and uh yeah all these things yeah 100 percent well congratulations for frontier and good luck on eva online it sounds honestly it sounds like a really exciting project a lot of fun to work on even if it means you can't sleep quite as well at night for a little while it's a little stressful it is like it is exciting i am proud of the team for deliberative frontier as i'm sure i'll be proud of of even they deliver there but i was fascinated by was that when i started starting looking into it i wanted kind of like the war stories and i didn't really find a lot of like people that had done it and i'm sure people a lot of people had they have done a very painful python 3 migration so my running theory was that nobody really wanted to talk about it after that and it's a traumatic

1:02:50experience it just let it go exactly all right well let's close out the show uh people who are maybe trying to accomplish the same thing they have some older code base they're trying to migrate even from maybe like old three to new three something like that give give them really quickly just some parting advice and also people interested in eva online christian we're gonna go first uh so parting advice for people that want to do this uh one of the things that i didn't mention is that we actually used a contract for a lot of the stuff uh called the racon digital uh they deserve a shout as well they did a really good job kind of migrating most of the python code and one of the reasons for it was also that developing a live game uh we kind of knew that frontier the development team like would it get immediate it derailed when they had to do some feature work and not supporting python 3 so like that was like one of the things that so i what i would probably argue there is like secure the resources like make them untouchable while you're doing this migration

1:03:53because once you start you have to finish it if you pause it it's almost wasted effort yeah that's the point yeah i was gonna actually say something similar that um because we've got one group working on on pushing ahead with the migration we still got a whole bunch of developers adding features to the game every day so about adding those features against r27 and so we're putting a lot of work into trying to make sure that they have uh but the new code that adding is as forward compatible as possible so a big thing we've rolled out this week is it's a new linter so we're getting and we're gonna progressively ratchet up the uh the strictness of it um so uh kind of there's kind of tools to kind of do code quality inspection i think are going to be really important for us going forward yeah 100% agree yeah um very much that uh i would add like you know the age old advice of like have really good test coverage it helps a lot but also um touch it up on what Jamie just said like if you have like a product that's life and you need to migrate it make sure

1:04:57that you do this in a separate branch but have constantly going ongoing migrations from the python two side to the python three side so that you discover all of these new pain points that may pop up as soon as possible um this helped us on on front here as well where there are sometimes things would come in that would clear that they're completely not python three compatible and then you know we could fix it up right away and the more frequent you do these integrations like the less painful they if you do it once a month oh my god like there was a period where we only did like every other week and uh we would basically spend a whole day uh just resolving all of the issues so do it frequently and do it often keep it small keep it small good advice do it in small small bite size bits yeah absolutely all right you guys thank you for being here everyone listening check out eve online eve online dot com very cool pro um very cool game very cool ecosystem check out the video i linked it says this is eve it's got some great graphics to inspire you as well bye everyone thank you thank you thank you thank you this has been another episode of talk python to me thank you to our sponsors be sure to check out what they're offering it really helps support

1:05:59the show this episode is sponsored by centuries sear if you're tired of debugging in the dark give sear or try there are plenty of AI tools that help you write code but centuries sear is built to help you fix it when it breaks visit talk python dot a fam slash century and use the code talk python 26 all one word no spaces or a hundred dollars in century credits and it's also brought to you by talk python courses course completion certificates are now live if you finished a course there's a certificate waiting for you on your account page right now download it as a pdf or add it to your linkedin profile with one click under licenses and certifications same section is your formal degrees visit training dot talk python dot a fam slash account to see what you've already earned if you or your team needs to learn python we have over 270 hours of beginner and advanced courses on topics ranging from complete beginners to acent code flask jingo htmx and even lm's best of all there's no subscription in sight browse the catalog at talk python dot a fam and if you're not

1:07:02already subscribed to the show on your favorite podcast player what do you waiting for just search for python in your podcast player we should be right at the top if you enjoy that geeky rap song you can download the full track the link is actually in your podcast player share notes this is your host michael kennedy thank you so much for listening i really appreciate it i'll see you next time bye