Discussion The Visited Stars cache

hchalkley

Senior Programmer
Frontier
Beta 2.2 introduced a new GalaxyMap filter, that indicates which stars you have visited: this is based on a cache file stored locally, but does not include historical data. The file is stored in the Local Appdata path, typically: c:\<username>\Appdata\Local\Frontier Developments\Elite Dangerous\<userid>\VisitedStarsCache.dat

I have been asked for information on the format of this file - but the file stores internal starsystem ID numbers which are not particularly suitable for access by third-party tools. We have instead added a mechanism to allow data to be imported into this cache.

If you write a file named ImportStars.txt into the same folder, with one star system name per line, then start the game, it will lookup the names and merge them into the cache. (If you have many thousands of star names in the file, it may take a few minutes to process)

Once it is finished, it will rename the file so it doesn't re-import them next time. It will report the total number of stars imported, the number of duplicates, and the number that failed lookup, via the message input window. This feature should be available in the next Beta release.
 
Beta 2.2 introduced a new GalaxyMap filter, that indicates which stars you have visited: this is based on a cache file stored locally, but does not include historical data. The file is stored in the Local Appdata path, typically: c:\<username>\Appdata\Local\Frontier Developments\Elite Dangerous\<userid>\VisitedStarsCache.dat

I have been asked for information on the format of this file - but the file stores internal starsystem ID numbers which are not particularly suitable for access by third-party tools. We have instead added a mechanism to allow data to be imported into this cache.

If you write a file named ImportStars.txt into the same folder, with one star system name per line, then start the game, it will lookup the names and merge them into the cache. (If you have many thousands of star names in the file, it may take a few minutes to process)

Once it is finished, it will rename the file so it doesn't re-import them next time. It will report the total number of stars imported, the number of duplicates, and the number that failed lookup, via the message input window. This feature should be available in the next Beta release.

Wonderful! I will start writing the export function on EDSM.
 
This is awesome! :D

No-Youre-Awesome-Meme-08.jpg
 
Awesome, we will can building a route through our systems:
1) Remove VisitedStarsCache.dat
2) Export our systems for each jump to ImportStars.txt and import this file
3) In galaxy map filtered only visited systems
4) Build route with filter

disadvantage - need restart game.
 
Awesome, we will can building a route through our systems:
1) Remove VisitedStarsCache.dat
2) Export our systems for each jump to ImportStars.txt and import this file
3) In galaxy map filtered only visited systems
4) Build route with filter

disadvantage - need restart game.

Intent: filter stars according to visited

Actual result: Allowing players to set custom routes

:D
 
Beta 2.2 introduced a new GalaxyMap filter, that indicates which stars you have visited: this is based on a cache file stored locally, but does not include historical data. The file is stored in the Local Appdata path, typically: c:\<username>\Appdata\Local\Frontier Developments\Elite Dangerous\<userid>\VisitedStarsCache.dat

I have been asked for information on the format of this file - but the file stores internal starsystem ID numbers which are not particularly suitable for access by third-party tools. We have instead added a mechanism to allow data to be imported into this cache.

If you write a file named ImportStars.txt into the same folder, with one star system name per line, then start the game, it will lookup the names and merge them into the cache. (If you have many thousands of star names in the file, it may take a few minutes to process)

Once it is finished, it will rename the file so it doesn't re-import them next time. It will report the total number of stars imported, the number of duplicates, and the number that failed lookup, via the message input window. This feature should be available in the next Beta release.

Question... Can we get an option (eg: In the Launcher or Game Menu or FDs Web Site) to create an ImportStars.txt by downloading all the stars we've ever visited directly from FD's servers? By all means make this as slow as you like, and only once a week or something, as it would be a very rarely used option.

1) This would then allow us to use the filter in the Galaxy Map for stars visited pre 2.2. ie: Nigh on two years worth of data some of would be missing!

2) If we ever need to re-install the game, we can get back all this historical data from FD.



Associated discussion - https://forums.frontier.co.uk/showt...nu-option-to-get-download-all-visited-systems



ps: And if/when a filter for "Discovered" is added, the same sort of process could be applied there too. ie: An option to download a complete list from the FD servers, and an import mechanism to update the game's local database.
 
Last edited:
This is really neat. I was slightly disappointed about not being able to retroactively add system names from before 2.2 with this feature, but being able to at least add key systems (like undiscovered Earthlikes I came across whilst exploring) that I've kept written down is a plus.

It would be nice to get the full list from the servers, of course.

I just worry that the idea to download all the visited stars from the servers would be prohibitively intensive, although perhaps it could be spread out over tiny potions over time to avoid choking the servers. (Just think about how many visited stars are listed under Statistics ingame and then think about how many CMDRs there are out there, making that number exponentially humongous...!)
 
Last edited:
I just worry that the idea to download all the visited stars from the servers would be prohibitively intensive, although perhaps it could be spread out over tiny potions over time to avoid choking the servers. (Just think about how many visited stars are listed under Statistics ingame and then think about how many CMDRs there are out there, making that number exponentially humongous...!)

Even if it only let you do it once a week, and then only trickled the data down? The data itself I suspect would be tens of K big for most users?

An example, I've been to just over 5000 systems, so if we assume each system means 25 characters of data? = 122K (ie: The size of an average JPG picture). That's even without compression or anything clever.

Personally I think the only hurdle is the request on FDs servers for, "read all CMDR XXXX visited systems", and it coming back with X thousand hits...


Another alternative: You request the download (in Menu, Launcher or Website) and it then runs it/produces it whenever convenient and emails it to you (zipped). You're only allowed to do this once a week etc...
 
Last edited:
Beta 2.2 introduced a new GalaxyMap filter, that indicates which stars you have visited: this is based on a cache file stored locally, but does not include historical data. The file is stored in the Local Appdata path, typically: c:\<username>\Appdata\Local\Frontier Developments\Elite Dangerous\<userid>\VisitedStarsCache.dat

I have been asked for information on the format of this file - but the file stores internal starsystem ID numbers which are not particularly suitable for access by third-party tools. We have instead added a mechanism to allow data to be imported into this cache.

If you write a file named ImportStars.txt into the same folder, with one star system name per line, then start the game, it will lookup the names and merge them into the cache. (If you have many thousands of star names in the file, it may take a few minutes to process)

Once it is finished, it will rename the file so it doesn't re-import them next time. It will report the total number of stars imported, the number of duplicates, and the number that failed lookup, via the message input window. This feature should be available in the next Beta release.

Ok cranked up EDDiscovery copied all the system names, made the file and stuck it in the right place ,,,re-checked this post,,,,,,,

By next beta release,,, do you mean Guardians 2.2 Beta 4?

BTW folks worried re file size I've visited about 22,000 systems and the file was only 375 k.
 
So as regard populating the local visited systems with all systems ever visited, it would surely seem:-
  • There is already an import option suitable for purpose (ie: to populate the local visited database)
  • FD have (in the past) produced a list of all visited systems. This could clearly easily be manipulated into the appropriate format for such an import.
  • The resultant file is not large. I'd estimated 25 bytes per system, and actual examples (see above) seem to suggest it's less than that. ie: We're typically talking about tens/hundreds of K for the file size.
Conclusion:-
  • Seemingly FD need only give us a means to re-access/download the visited data (again), and this will work. And they could do that as restrictively as they like. eg: Just one request a week, given how infrequently it would be required.


Associated discussion - https://forums.frontier.co.uk/showt...nu-option-to-get-download-all-visited-systems
 
Last edited:
Even if it only let you do it once a week, and then only trickled the data down? The data itself I suspect would be tens of K big for most users?
An example, I've been to just over 5000 systems, so if we assume each system means 25 characters of data? = 122K (ie: The size of an average JPG picture). That's even without compression or anything clever.
Personally I think the only hurdle is the request on FDs servers for, "read all CMDR XXXX visited systems", and it coming back with X thousand hits...
Another alternative: You request the download (in Menu, Launcher or Website) and it then runs it/produces it whenever convenient and emails it to you (zipped). You're only allowed to do this once a week etc...

But server to-and-from queries double that again, in terms of bandwidth, no? And you are assuming that information is indexed and readily accesible & ready-to-go...and the server query itself might require more data than what we're trying to get from it, as well, at least to my foggy knowledge.

Simply put, I don't think it's that simple!
 
I just had a thought though...what if we could request the list of stars we've visited prior to 2.2, through contacting Support and using the ticket system, with it being spelled out clearly to us players that it may take time to process the request?

That way, Frontier could handle things on their time at their pace instead of potentially drowning out the servers with requests for all that information, and additionally only have to worry about players who ask for it, as opposed to potentially dealing with all CMDRs everywhere.
 
Last edited:
If you write a file named ImportStars.txt into the same folder, with one star system name per line, then start the game, it will lookup the names and merge them into the cache. (If you have many thousands of star names in the file, it may take a few minutes to process)

Once it is finished, it will rename the file so it doesn't re-import them next time. It will report the total number of stars imported, the number of duplicates, and the number that failed lookup, via the message input window. This feature should be available in the next Beta release.

89HANHg.gif
 
But server to-and-from queries double that again, in terms of bandwidth, no? And you are assuming that information is indexed and readily accesible & ready-to-go...and the server query itself might require more data than what we're trying to get from it, as well, at least to my foggy knowledge.

Simply put, I don't think it's that simple!

See some of the posts above (eg: #12 for a summary).

It seems at one point FD allowed the visited systems to be returned by a call. Now I suspect they removed this as it was intensive. But I suspect it was intensive if loads of external applications were calling it when ever they wanted to. I'm suggesting giving us that same access again, but simply restricting/limiting it.

ie: Rather than external application calling for that data potentially over and over, when ever they want, allow it to be done just once a week etc.

In truth I suspect most users will at most use the feature once to do an initial population/update. And then maybe again later if they ever have to re-install the game.


But in short... The import is there. The download was there (at one point)... It's looking more and more feasible IMHO... The only hurdle would seem to be FD allowing access to that call again in some shape or form.
 
Last edited:
See some of the posts above (eg: #12 for a summary).

It seems at one point FD allowed the visited systems to be returned by a call. Now I suspect they removed this as it was intensive. But I suspect it was intensive if loads of external applications were calling it when ever they wanted to. I'm suggesting giving us that same access again, but simply restricting/limiting it.

ie: Rather than external application calling for that data potentially over and over, when ever they want, allow it to be done just once a week etc.

In truth I suspect most users will at most use the feature once to do an initial population/update. And then maybe again later if they ever have to re-install the game.


But in short... The import is there. The download was there (at one point)... It's looking more and more feasible IMHO... The only hurdle would seem to be FD allowing access to that call again in some shape or form.

Wouldn't it be easier to let the whole thing trickle down to all players over an extended time period (weeks or even a few months). Those who care will of course take a backup. This way ED can adjust the server load themselves and optimise the indexing with this in mind.

There might be more development involved here so possibly not a viable approach.
 
Wouldn't it be easier to let the whole thing trickle down to all players over an extended time period (weeks or even a few months). Those who care will of course take a backup. This way ED can adjust the server load themselves and optimise the indexing with this in mind.

There might be more development involved here so possibly not a viable approach.

Easier for us... Not for FD I suspect.

Going down the download/import approach the puzzle pieces are already there, and fitting! - The import is there already! The download was there, so just needs to be opened up to us again is some guise. Done!

And it does then mean for those more technically minded folks they can fiddle too!
 
Last edited:
Easier for us... Not for FD I suspect.

Going down the download/import approach the puzzle pieces are already there, and fitting! - The import is there already! The download was there, so just needs to be opened up to us is some guise. Done!

Yeah, I suspect you are right. I'm lucky though. With the exception of a few weeks, I have verbose logs all the way back to December 2014.
 
Yeah, I suspect you are right. I'm lucky though. With the exception of a few weeks, I have verbose logs all the way back to December 2014.

Going through all of those just to sift out the system names and add them to this cache manually sounds like a lot of work to me...more power to you if you're up for it though!
 
Top Bottom