Train new model

[yguezpa]

Hello,

Thanks for this impressive work.
If I want to train a model on completely different type of data, what should I do ? Can I use the .jar trained on Ember ?

Thanks

[rjzak]

You can create a new set of Bloom filters using this command java -cp AutoYara-1.0-SNAPSHOT.jar edu.lps.acs.ml.autoyara.Bytes2Bloom based on any data you want. It's important to make sure you have a lot of the data so AutoYara can figure out what's important in your target set, and probably have Bloom filters for different sizes of n.

[yguezpa]

hey @rjzak
Thanks for the quick answer!
1 - Do you have a tutorial that I can follow with all the steps to train a model from scratch ?

2 - How should I give the malicious and benign to the command java -cp AutoYara-1.0-SNAPSHOT.jar edu.lps.acs.ml.autoyara.Bytes2Bloom. I see there is a -i argument. Should it be the benign or malicious (or both) ?

3 - I am trying to run like this but I get this exception. Could you tell me what's the problem ?

Thanks

[rjzak]

We don't have any tutorials or great documentation unfortunately. Essentially you'll have two directories, one with benign bloom filters, one with malicious bloom filters.

Create the bloom filter file on a per-ngram size, per goodware/malware basis:

java -cp AutoYara-1.0-SNAPSHOT.jar edu.lps.acs.ml.autoyara.Bytes2Bloom -i ~/Documents/LiveSamples/PDF/Goodware/ -o benign/ -k 100000 -n 8
AutoYara version 1.0-SNAPSHOT, compile date: 2022-08-23 16:06:35
Starting 8-grams of 2200 files...
Finding top-k hashes
$ ls benign/
Goodware_8.bloom

• -k: Keep the top 100,000 most common ngrams
• -n: Collect 8-grams
• -o: Output directory, where the file name is the outputDir/sourceDirName_N.bloom

So collect bloom filter files for different sizes of n and k, keeping malware and benign separate (output in separate directories). Then have AutoYara use those directories when creating the Yara rule for a target collection.

[yguezpa]

hey @rjzak,

Thanks for the help. I managed to create some bloom filters for malicious and benign.
Then I need to run java -jar AutoYara.jar but how should I use it ?

Let say that I want to create YARA rules that detect malicious files, should I run this command ?

jav -jar {AUTOYARA_JAR} -b {benign_bloom_folder} -m {malicious_bloom_folder} --fp-dirs {benign_paths_file} --input-dir {malicious_paths_file} -o {results_rule} -fpb {fp_rate} -fpm {fp_rate}

[rjzak]

Yes, that's it. But I think you only need -b {benign_bloom_folder} -m {malicious_bloom_folder} --input-dir {malicious_paths_file} -o {results_rule}. The changes to the -fpb and -fpm I think should be tweaks after you analyze the resulting Yara rule, or if you know ahead of time that your data or intended target is odd in some way.

[yguezpa]

and --fp-dirs {benign_paths_file} ?
Without this how it will detect YARA rules that lead to false positive

[rjzak]

Maybe that's a newer feature I wasn't aware of. Follow the docs and code before listening to me :)