Datalog File Trimming Software!

[Brennan]

V1.1.2 released and attached! TL;DR Attached is a program ("DaFT", Datalog File Trimmer) I made that will filter and highlight your datalogs for you!


WHAT YOU CAN DO WITH THIS:

1. Filter logs
   • Choose min/max values to filter out rows from a log. This can be done on up to 10 columns.
2. Highlight logs
   • Choose any number of: highlight column, color, any conditions you wish to specify.

The default behavior is to

• Filter out any row with Throttle Position < 68
• (Default highlight information is shown in the screenshot)

You can change the filters and highlights to whatever you want and save them as your defaults.

Background:
A little while ago I got an AP and started logging. I got tired of filtering them down to just the WOT run and also of highlighting the columns I actually cared about. Being a software engineer, I decided to make a little piece of software to do this for me. Then I figured other people might appreciate it as well, so I cleaned it up a bit and now here it is.

To install, just unzip the attached ZIP folder and run setup.exe. You can then run the program from the Start menu (it's not in a folder and is called "DaFT").

I've only used this with AP logs, but, in theory, you should be able to use this to filter and highlight any CSV file. Note that minimum, maximum, and average of each column is also included if you choose to output to XLSX (see the second sheet in the XLSX file). Oh yeah, and don't forget to click Help -> About .

Don't have Excel to view the output XLSX files? Download Open Office.

Screenshot:


Sample Output (using "factory default" filters and highlights):

[gearhead750]

Domo Arigato kind sir you rock!

[symkaz]

Just downloaded and tested it out. Thanks for sharing.

[Brennan]

Originally Posted by symkaz Just downloaded and tested it out. Thanks for sharing.

Happy to be able to give something back to the community that has provided me with so much information. If you have any issues with it, or have any suggestions, you can post 'em up here or send an email to the address in the about window.

[CosmicArkie]

I just wish you'd come up with this while I was still logging and tuning.

With all the zoom zoom boom going on around here lately, I'm too chicken to
drive the thing.

Thanks a lot.

J

[Etipp98]

Sweet Program! DL'd!

[leon78]

Originally Posted by Brennan I got tired of filtering them down to just the WOT run

So you are the guy who logs your 20min drive to work in stop and go traffic...Nah...Good shit man….thanks. This will help out guys trying to trouble shoot shit here.

[Crarrs]

@Raider

Can we either sticky this thread or add it to the ECU/Tuning Sticky?

[Brennan]

Originally Posted by leon78 So you are the guy who logs your 20min drive to work in stop and go traffic...Nah...Good shit man….thanks. This will help out guys trying to trouble shoot shit here.

LOL. I'd actually like to add the ability to take a single datalog input and output multiple files from it, based on the filters. So, for instance, you could do multiple WOT runs without stopping logging (and presumably with normal driving between) and DaFT would split it out into separate files for you.

I'm not sure how big of a single log the AP will let you store though. And this wouldn't be desirable if you had a multi-gear log, but you could just disable the option for that log. I'm probably overthinking this...

Originally Posted by Raider Stuck in ECU Tuning Section. @Brennan, look for MSF Bucks

Thank you kind sir!

[Raider]

Stuck in ECU Tuning Section. @Brennan, look for MSF Bucks

[nindoja]

I don't know much about the XLSX format, other than it's a zip file of XML basically, but could you add in conditional formatting?

I'd love to be able to filter to WOT logs and then have conditional formatting set for some columns (KR > 2 = yellow, KR > 5 = red, HPFP pressure < 1600 = red, etc.). Come to think of it, I've done this before just not with CSV files.

Not sure if you could get the UI right or if it would overcrowd it, but it'd be sweet.

Also, this would be sweet to combine with @Enki's Kalibrator program. Do a bunch of long logs to feed into that, but use this for when you only want to look at the WOT sections.

[Enki]

Originally Posted by nindoja I don't know much about the XLSX format, other than it's a zip file of XML basically, but could you add in conditional formatting? I'd love to be able to filter to WOT logs and then have conditional formatting set for some columns (KR > 2 = yellow, KR > 5 = red, HPFP pressure < 1600 = red, etc.). Come to think of it, I've done this before just not with CSV files. Not sure if you could get the UI right or if it would overcrowd it, but it'd be sweet. Also, this would be sweet to combine with @Enki's Kalibrator program. Do a bunch of long logs to feed into that, but use this for when you only want to look at the WOT sections.

My app actually breaks it all down by the core voltage and trim level and is meant more for calculating MAF curves. Also, I still need an alpha tester or two to send me the proper stuff before I release it to the tuning groups for further review.

Edit:
It also doesn't like anything other than raw AP logs at this time, so is fairly limited.

[Brennan]

Originally Posted by nindoja I don't know much about the XLSX format, other than it's a zip file of XML basically, but could you add in conditional formatting? I'd love to be able to filter to WOT logs and then have conditional formatting set for some columns (KR > 2 = yellow, KR > 5 = red, HPFP pressure < 1600 = red, etc.). Come to think of it, I've done this before just not with CSV files. Not sure if you could get the UI right or if it would overcrowd it, but it'd be sweet. Also, this would be sweet to combine with @Enki's Kalibrator program. Do a bunch of long logs to feed into that, but use this for when you only want to look at the WOT sections.

What you want is exactly what it does, with one exception: multiple colors in a single column are not currently available (should be soon). However, you can set it up to color anything in a range of values for a particular column, and you choose the min/max/color.

This same thing applies to the filtering: you get to set whatever filters you want; it is not limited to only trimming throttle position to get a WOT run.

I'm updating the original post to be clearer and show the default behavior.

[nindoja]

Originally Posted by Brennan What you want is exactly what it does, with one exception: multiple colors in a single column are not currently available (should be soon). However, you can set it up to color anything in a range of values for a particular column, and you choose the min/max/color. This same thing applies to the filtering: you get to set whatever filters you want; it is not limited to only trimming throttle position to get a WOT run. I'm updating the original post to be clearer and show the default behavior.

Oh, ok. I thought this would just trim down the log that would then get imported to Excel.

Originally Posted by Enki My app actually breaks it all down by the core voltage and trim level and is meant more for calculating MAF curves. Also, I still need an alpha tester or two to send me the proper stuff before I release it to the tuning groups for further review. Edit: It also doesn't like anything other than raw AP logs at this time, so is fairly limited.

Yeah, I know it's different, but if I use both of these apps I no longer have to do separate logs for WOT pulls vs daily driving logs. And I can quickly separate out my WOT pulls.

I'll hopefully get some good logs done this week for you Enki, sorry for the delay.

[Brennan]

Originally Posted by Enki It also doesn't like anything other than raw AP logs at this time, so is fairly limited.

If we're talking about running a log through DaFT and then through your program for a MAF cal, DaFT supports output to CSV (no highlighting), so it would look just like a raw datalog. Also, I just nabbed some logs for a MAF cal, and would be happy to send 'em to you or whatnot if that's what you need.

[Enki]

Originally Posted by Brennan If we're talking about running a log through DaFT and then through your program for a MAF cal, DaFT supports output to CSV (no highlighting), so it would look just like a raw datalog. Also, I just nabbed some logs for a MAF cal, and would be happy to send 'em to you or whatnot if that's what you need.

Right now the app requires a log format exactly like that which is output from the AP (which isn't actually even a CSV file, but a TXT file (tab delimited vs comma delimited).

As for the log file, the requirements for the alpha test are listed here:
Enkiworks Kalibrator

[Brennan]

Any interest in updates to the software? I haven't worked on it since I released it, mainly because I don't want to waste time if there's little interest, and I've been busy with other things.

[nindoja]

@Brennan: Just got around to using this and I'm impressed. Any chance we could get more advanced highlighting/filter rules?

For example, I'd like to only trigger HPFP Act. pressure < 1600 when APP > 90%?

Just saw the min/max/avg sheet that gets output when doing xlsx, freaking awesome! I usually look for max KR, min FP, so that's an awesome feature.

[Brennan]

Originally Posted by nindoja @Brennan: Just got around to using this and I'm impressed. Any chance we could get more advanced highlighting/filter rules? For example, I'd like to only trigger HPFP Act. pressure < 1600 when APP > 90%? Just saw the min/max/avg sheet that gets output when doing xlsx, freaking awesome! I usually look for max KR, min FP, so that's an awesome feature.

That's a good idea. I think I can make that happen, with a bit of an interface change. The other things I'd like to include in this first update:

1. Freeze top row in XLSX.
2. Make min/max/avg calculated by Excel so that if you modify the datalog sheet, the values are recalculated...
3. Multiple highlights in single column. For example, you could set KR > 0.0 yellow, and KR > 2.0 red. Currently, DaFT will only use one of these.
4. Shouldn't need write access to original CSV files... Currently DaFT cannot process them if they're being used by another process (e.g. open in Excel).

As for a time frame, I'm not really sure. I'll see what I can get done this week.

[Brennan]

Originally Posted by Brennan That's a good idea. I think I can make that happen, with a bit of an interface change. The other things I'd like to include in this first update: 1. Freeze top row in XLSX. 2. Make min/max/avg calculated by Excel so that if you modify the datalog sheet, the values are recalculated... 3. Multiple highlights in single column. For example, you could set KR > 0.0 yellow, and KR > 2.0 red. Currently, DaFT will only use one of these. 4. Shouldn't need write access to original CSV files... Currently DaFT cannot process them if they're being used by another process (e.g. open in Excel). As for a time frame, I'm not really sure. I'll see what I can get done this week.

I have 2-4 done, along with the suggested "advanced highlighting/filter rules", but I've only done it for highlighting.

To be honest, I don't see the need for adding this same thing for filtering since the filter specifications are all "AND"ed together anyway. But please let me know if I'm overlooking something in thinking that. Screenshot of the new highlight tab is format:




#1 should really only take a half hour tops, and I entirely plan on releasing V2 this weekend. So far, the entirety of the software, including these V2 updates, has probably taken me about 40 hours.

Please let me know what "factory defaults" you guys would like to see, although changing those forever is just a simple click of the "Save" button. I'll update the original post and add a post in the thread when I get V2 all squared up.

Cheers,
Brennan

[backward_0]

subbed for knowledge and updates. Good work dude!

[nindoja]

looking awesome @Brennan. Agreed with the filters part, so no worries there. Can't wait to try V2.

[backward_0]

very cool looking forward to the update

[Brennan]

Original post updated with version 1.1!

VERSION NOTES:
1. A log is now created in a "logs" directory every time DaFT is run. Logs older than one week will automatically be deleted.
2. Top row is now frozen in XLSX.
3. Min/max/avg are now calculated using Excel formulas, so they will be correct if you modify the datalog sheet.
4. DaFT no longer needs write access to the original CSV in order to process it.
5. You can now have multiple highlights for a single column (with different colors/conditions for highlight).
6. Advanced highlighting rules; highlights are now unique by column name and color instead of just column, and up to five conditions may be specified. A condition consists of a column, min value for that column, and a max value for that column. You can also specify whether the highlight column's value has to meet "any" or "all" of the specified conditions in order for the highlight color to be applied.
7. Fixed an issue with filters getting confused between Boost and Boost Air Temp.
8. Filters are now inclusive of the min/max specified. They were exclusive in version 1.0.0.0.

[hnda etr]

Good stuff! Haven't downloaded yet, but definitely will!

Since people are throwing out "wishes" how about adding the ability to graph your logs using Excel's graphing (or pivot chart?) function... Would need to be able to select values for each axis as well as pick and choose what values to include in graph. Also, different color for each value...

Just throwing it out there ;-)

[Brennan]

Originally Posted by hnda etr Good stuff! Haven't downloaded yet, but definitely will! Since people are throwing out "wishes" how about adding the ability to graph your logs using Excel's graphing (or pivot chart?) function... Would need to be able to select values for each axis as well as pick and choose what values to include in graph. Also, different color for each value... Just throwing it out there ;-)

Sorry I took so long to respond to this. If this is something that can just be set up in Excel and is based on whatever values/columns are present in the sheet it shouldn't be very difficult to do. I'll have to look into it when I have some time .

[hnda etr]

Windows 7 x64:

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.UnauthorizedAccessException: Access to the path 'logs' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, DirectorySecurity dirSecurity)
at System.IO.Directory.CreateDirectory(String path, DirectorySecurity directorySecurity)
at DaFT.Logger.getWriter()
at DaFT.Logger.logError(String msg)
at DaFT.DaFTForm.browseButton_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
mscorlib
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5456 (Win7SP1GDR.050727-5400)
CodeBase: file:///C:/Windows/Microsoft.NET/Fra...7/mscorlib.dll
----------------------------------------
DaFT
Assembly Version: 1.1.0.0
Win32 Version: 1.1
CodeBase: file:///C:/Program%20Files%20(x86)/DaFT/DaFT.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5460 (Win7SP1GDR.050727-5400)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...dows.Forms.dll
----------------------------------------
System
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5456 (Win7SP1GDR.050727-5400)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...089/System.dll
----------------------------------------
System.Drawing
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5462 (Win7SP1GDR.050727-5400)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...em.Drawing.dll
----------------------------------------
System.Core
Assembly Version: 3.5.0.0
Win32 Version: 3.5.30729.5420 built by: Win7SP1
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...ystem.Core.dll
----------------------------------------
System.Data
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5420 (Win7SP1.050727-5400)
CodeBase: file:///C:/Windows/assembly/GAC_64/S...ystem.Data.dll
----------------------------------------
System.Xml
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.5420 (Win7SP1.050727-5400)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...System.Xml.dll
----------------------------------------
Microsoft.Office.Interop.Excel
Assembly Version: 14.0.0.0
Win32 Version: 14.0.4756.1000
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...erop.Excel.dll
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:



When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

[Brennan]

Originally Posted by hnda etr System.UnauthorizedAccessException: Access to the path 'logs' is denied.

Try running DaFT as an administrator. The "logs" folder it tries to create would be "C:\Program Files (x86)\DaFT\logs" on an x64 machine. It looks like the user you were logged in as did not have permission to create that directory for whatever reason (i.e. did not have write access to the "C:\Program Files (x86)\DaFT" folder).

Very likely this would correct the issue: check the "Security" tab on the "C:\Program Files (x86)\DaFT" folder properties, and enable write access for the user/user group you wish.

Could this just be a result of choosing to install "Just for this user" as opposed to for "Everyone"? Maybe. Perhaps I should change the default to "Everyone"...

In any event, please let me know if that works for you. I hadn't tested anything with multiple user accounts/permissions. Thanks.

[memin]

Originally Posted by Brennan Try running DaFT as an administrator. The "logs" folder it tries to create would be "C:\Program Files (x86)\DaFT\logs" on an x64 machine. It looks like the user you were logged in as did not have permission to create that directory for whatever reason (i.e. did not have write access to the "C:\Program Files (x86)\DaFT" folder). Very likely this would correct the issue: check the "Security" tab on the "C:\Program Files (x86)\DaFT" folder properties, and enable write access for the user/user group you wish. Could this just be a result of choosing to install "Just for this user" as opposed to for "Everyone"? Maybe. Perhaps I should change the default to "Everyone"... In any event, please let me know if that works for you. I hadn't tested anything with multiple user accounts/permissions. Thanks.

i got the same message and i selected everyone on the set up!! I did those steps but same outcome, not working for me on winows 7.

[Brennan]

Originally Posted by memin i got the same message and i selected everyone on the set up!! I did those steps but same outcome, not working for me on winows 7.

@memin
@honda etr
Well, that blows nuts. Try the attached version. A "DaFT\logs" folder should now be created in "C:\Users\\Documents" as recommended.

The odd thing is that I tested DaFT on Windows 7 x64, though it was under an administrator account.

Anyway, please let me know how this works out. Should only take a few seconds to test. Thanks!


Edit: Removed attachment to attach in the original first post. See first post for v1.1.1.

[Karmat]

Excellent work I'll give this a shot when I do some logs.

Sent from my DROID RAZR using Tapatalk

[Enki]

Might I suggest writing the logs to the user profile, specifically the documents folder?

[Enki]

Just tested for you (as local admin, but with uac enabled) in 7x64. Installed via doubleclicking the MSI (skipping the exe), prompted to allow admin access, and it installed without issue. Running also worked just fine.

Out of curiosity, are you hard-coding the path for NT os or are you using the %userprofile% system variable?

[Brennan]

Originally Posted by Enki Just tested for you (as local admin, but with uac enabled) in 7x64. Installed via doubleclicking the MSI (skipping the exe), prompted to allow admin access, and it installed without issue. Running also worked just fine. Out of curiosity, are you hard-coding the path for NT os or are you using the %userprofile% system variable?

With the change I just made (C#):

string logDir = Environment.GetFolderPath(Environment.SpecialFolder.Personal) + "\\DaFT\\logs";

Personal here is the same as "My Documents".

[CosmicArkie]

Loads and works well on W7x32.

Thanks.

[Wicked]

This is going to be a life saver when it comes to reading my logs. Bravo sir!

[fortressofcomfort]

Nice Microsoft Stackin Skillz brah!

What 3.5* spark adv at 4500rpms? Is that all she'll take or are they kinda like demo numbers?

[Brennan]

Originally Posted by fortressofcomfort What 3.5* spark adv at 4500rpms? Is that all she'll take or are they kinda like demo numbers?

The low HPFP pressure and the 2+ KR were "demo numbers", I think the rest was the actual log. Must have been stage 1 OTS for timing. Looks like before I had a fuel pump and before I touched timing, boost was at 16.5 cause I was scurred of ZZB. I just finished tweaking timing to 8* at 4500 RPM on my current map yesterday. Was up to 9* with a wee bit of consistent knock on pump 93, knock disappeared with octane booster, I dropped a degree to 8 and will leave it there .

[fortressofcomfort]

I was just messing w you man, yeah I run 9* at 4.5k, close enough.

REALLY I posted because I was impressed with your programming skills. What are you using? VS2010? C# or VB or ?

[Brennan]

Originally Posted by fortressofcomfort I was just messing w you man, yeah I run 9* at 4.5k, close enough. REALLY I posted because I was impressed with your programming skills. What are you using? VS2010? C# or VB or ?

Haha, why thank you. DaFT was written using C# in VS 2008 professional (got it for free while still in college); I'd bet VS express (free) would have worked just as well. Did the majority of it in free time over a week (live-in girlfriend was less than amused). I was originally just going to write a script to trim CSVs down to WOT portions, then I decided something a bit more powerful might be handy and I wanted to brush up on C# (I only ever script or write C++ at work, and I wrote my last at-home program in Java). At that point, I figured the forum might also find it useful so I tried to make it presentable and easy to use.

I'm just glad it's helping people out . If you have anything you'd like to see changed or added, let me know! And I might eventually get around to adding graphs to a spreadsheet tab as suggested...