Discussion Example wanted to read journal file (C++, python or js)

I'm "growing one", being brave and asking for help.

To date, I've nutted things out myself by looking on line, or trial and error.
In the last 12 months in particular, I've almost (95% as far as I can tell) mastered Thrustmaster's TARGET script, learnt to code in python and java-script (beginner level) and most recently C++ (somewhere between beginner and basic DIY hack)... all because I'm fascinated in getting my TARGET script to "know" what's going on in game.

Using TARGET Script, I can now read status.json and extract key-values on demand.
I can also use an external c++ program to send the Target Script just the Flags key value everytime it changes (using a GameCallback function within Target).

My next challenge is to start reading/extracting richer information from whichever journal file is currently active.

This has proven beyond my meager talent to do.
I know others have mastered this and am asking someone/anyone who knows how best to do this, to give me a leg up.

I prefer a C++ example as I feel I have a better handle on this than python or javascript at this stage.

Any takers?

Clicker
 
Hi @Robert Maynard ,

Thanks for replying.
I'm using rapidjson with C++ at present to be able to process what I want out of the status.json file.
As you know, this is a single line json and a breeze to just open/read/process.

What I'm really after here is;
  • Method on how to determine which journal file within the folder is the active one
    • I'm guessing I need to construct currentdate+time as a starting point then search the folder for journal.datetime.01.log
    • search windows folders for files is the bit I need to learn
  • Method on best way to read the latest entry in the file.
    • again, once I have the right file, is there a shortcut way to go straight to the latest entry without reading in every line?
    • or, just focus on the "event" I'm after and finding the latest entry for that
I can then do the rest via interrogating the "event" key.

Cheers
Clicker
 
Last edited:
The latest file can probably be best given to you by the operating system. Depending on the file system API, you might be able to query for the file last modified. If the file gets too long, I think there will be an event telling you the new file. If the game is restarted while your program is running, you'd need to re-query for the latest file.

Getting the latest entry requires either a hook to the filesystem that gives you notice once something has been appended or requires pulling over and over.

These are not precise solutions, but without having a fixed programing language (you hinted in your last post that you were using C++) it's hard to give accurate advice. I hope you know what to look for now, though.
 

Robert Maynard

Volunteer Moderator
What I'm really after here is;
  • Method on how to determine which journal file within the folder is the active one
    • I'm guessing I need to construct currentdate+time as a starting point then search the folder for journal.datetime.01.log
    • search windows folders for files is the bit I need to learn
In answer to this bit:

Code:
inline TimeClass TimeClassFromString(std::string time_string)
{
    FILETIME fTime = {};

    if (time_string.length() != 14)
    {
        return fTime;
    }

    SYSTEMTIME sTime = {WORD(atoi(time_string.substr(0, 4).c_str())), WORD(atoi(time_string.substr(4, 2).c_str())), 0, WORD(atoi(time_string.substr(6, 2).c_str())),
                        WORD(atoi(time_string.substr(8, 2).c_str())), WORD(atoi(time_string.substr(10, 2).c_str())), WORD(atoi(time_string.substr(12, 2).c_str())), 0};

    SystemTimeToFileTime(&sTime, &fTime);

    return TimeClass(fTime);
}

TimeClass TimeClassFromJournalFilename(std::string thisFile_name)
{
    std::string tempString = "20";

    if (thisFile_name.substr(0,8) == "Journal.")
    {
        tempString += thisFile_name.substr(8,12);
    }
    else
    {
        if (thisFile_name.substr(0,12)== "JournalBeta.")
        {
            tempString += thisFile_name.substr(12,12);
        }
    }
    return TimeClassFromString(tempString);
}

std::string FindNewestJournal(std::string thisDirectory, int32_t thisPart)
{
    WIN32_FIND_DATA wfd;
    HANDLE hFind = INVALID_HANDLE_VALUE;

    std::string thisFile_name;

    std::string thisPart_string = NumToStrPadZ(thisPart,2);

    if (thisPart < 1)
    {
        thisPart_string = "??";
    }

    hFind = FindFirstFile((thisDirectory + "Journal*." + thisPart_string + ".log").c_str(),&wfd);

    if (hFind == INVALID_HANDLE_VALUE)
    {
        FindClose(hFind);
        return emptystring;
    }

    thisFile_name = wfd.cFileName;
    TimeClass thisFile_time_from_name(TimeClassFromJournalFilename(thisFile_name));

    while (FindNextFile(hFind, &wfd) != 0)
    {
        TimeClass next_file_time_from_name(TimeClassFromJournalFilename(wfd.cFileName));

        if (next_file_time_from_name.cns() > thisFile_time_from_name.cns())
        {
            thisFile_name = wfd.cFileName;
            thisFile_time_from_name = next_file_time_from_name;
        }
    }

    FindClose(hFind);

    return thisFile_name;
}
.... where:
1) TimeClass is a class I wrote for handling Windows 64-bit time used in FILETIME (units of 100ns since 01/01/1601);
2) .cns() returns the raw int64_t;
3) NumToStrPadZ returns a zero padded numeric string of the desired length.

Other points to note regarding the approach used in my app:
1) The selected journal is opened and read to the end, populating an ever increasing array of journal lines. After each read, any new lines are parsed.
2) Every [insert period here], the journal file is read again (from the previously encountered "end") - and any new lines added and parsed.
3) My app checks for a newer journal every fifteen seconds - to allow the app to continue in the event of a game crash (rather than waiting forever for a new line in an obsolete journal).
 
Last edited:
If you just want to get the latest events as they come in then you could look at writing a plugin for EDMC - then let EDMC worry about all the file access - plugins just get passed the events as they come in https://github.com/Marginal/EDMarketConnector/blob/master/PLUGINS.md . Plugins also get access to the status.json info as that changes.

ofc EDMC is python, and if you want to display stuff inside EDMC it is ... somewhat arcane - but passing stuff elsewhere should be easy.
 
Method on how to determine which journal file within the folder is the active one
  • I'm guessing I need to construct currentdate+time as a starting point then search the folder for journal.datetime.01.log
  • search windows folders for files is the bit I need to learn
You can use python's glob module to obtain the same results with very low effort:
Python:
logs = glob.glob("/mnt/c/Users/username/Saved Games/Frontier Developments/Elite Dangerous/Journal.*.log")
logs.sort()
last = logs[-1]
This will basically enumerate all the files that match the specified pattern Journal.*.log in the journals folder (note I'm using Unix style paths because I'm running into WSL) and sort them (glob returns them in an arbitrary order) so that the last is the most recent one. FDev was kind enough to format the filenames so that it's sufficient to sort them alphabetically to be sure to get the last one. Alternatively, you can sort them according to the modification time of the file (if you don't touch them manually, the active one will be the latest to be modified by the client itself) like this:
Python:
logs = glob.glob("/mnt/c/Users/username/Saved Games/Frontier Developments/Elite Dangerous/Journal.*.log")
logs.sort(key=lambda x: os.path.getmtime(x))
last = logs[-1]
Method on best way to read the latest entry in the file.
  • again, once I have the right file, is there a shortcut way to go straight to the latest entry without reading in every line?
  • or, just focus on the "event" I'm after and finding the latest entry for that
Unfortunately, files are sequential objects so you have to go through them somehow, either by reading or seeking. In this particular case, however, you have one big advantage: you want to read the last entry, and FDev decided to put one entry per line in the journal file. So if you really feel you want to go through the hassle (these logs are not huge) you can go to the end of the file using seek:
Python:
with "/mnt/c/Users/user/Saved Games/Frontier Developments/Elite Dangerous/Journal.191125154137.01.log" as f:
    f.seek(0, 2) # move of 0 from whence = 2 = SEEK_END
and then start reading backwards one character at a time, until you find a newline, then the next, etc.
This is a bit painful to code, I would warmly suggest just reading the whole file one line at a time, until you get to the last. In python, if you're not worried to keep the whole file in memory, it's as easy as:
Python:
with open("/mnt/c/Users/username/Saved Games/Frontier Developments/Elite Dangerous/Journal.191125154137.01.log", "r") as f:
    lines = f.readlines()
    print lines[-1]
Which can be wildly optimized in so many different ways that it's pointless to discuss here.

If you then want to go to the new line and wait for new lines to be written, to parse the file in real time, you can use something like this: https://gist.github.com/Haujilo/03a8473149643ebf7e32
 
Top Bottom