Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Erros when generating distance file #138

Open
DigOrDog opened this issue Sep 27, 2023 · 5 comments
Open

Erros when generating distance file #138

DigOrDog opened this issue Sep 27, 2023 · 5 comments

Comments

@DigOrDog
Copy link

SETUP:

  1. OS:ubuntu 20.04
  2. RAM:250G
  3. aflgo:commit 1b81e7c
  4. Fuzz Taregt:llvm-dis(llvm-15.0.4)
  5. BBtargets.txt
llvm/tools/llvm-dis/llvm-dis.cpp:108
llvm/tools/llvm-dis/llvm-dis.cpp:109
llvm/tools/llvm-dis/llvm-dis.cpp:110

Steps To Reproduce

  1. run AFLGo building script to configure.
  2. download llvm-15.0.4
# download and tar
wget https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-15.0.4.tar.gz
tar -zxvf llvmorg-15.0.4.tar.gz
mv llvm-project-llvmorg-15.0.4 llvm-project
export SUBJECT=$PWD/llvm-project

3.Set targets

# Setup directory containing all temporary files
mkdir temp
export TMP_DIR=$PWD/temp

BBtargets.txt
llvm/tools/llvm-dis/llvm-dis.cpp:108
llvm/tools/llvm-dis/llvm-dis.cpp:109
llvm/tools/llvm-dis/llvm-dis.cpp:110

# Print extracted targets. 
echo "Targets:"
cat $TMP_DIR/BBtargets.txt
  1. generate CG and intra-procedural CFGs from the subject.
# Set aflgo-instrumenter
export CC=$AFLGO/instrument/aflgo-clang
export CXX=$AFLGO/instrument/aflgo-clang++

# Set aflgo-instrumentation flags
export COPY_CFLAGS=$CFLAGS
export COPY_CXXFLAGS=$CXXFLAGS
export ADDITIONAL="-targets=$TMP_DIR/BBtargets.txt -outdir=$TMP_DIR -flto -fuse-ld=gold -Wl,-plugin-opt=save-temps"
export CFLAGS="$CFLAGS $ADDITIONAL"
export CXXFLAGS="$CXXFLAGS $ADDITIONAL"

export LDFLAGS=-lpthread
# Build only LLVM
pushd llvm-project
mkdir build
pushd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release 
-DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX"
-DCMAKE_CXX_FLAGS="$ADDITIONAL" -DCMAKE_C_FLAGS="$ADDITIONAL" 
-DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" 
../llvm
ninja
  1. Generate distance file. Firstly we need to clean up BBnames.txt and BBcalls.txt, otherwise distance_calculator may fail.
cat $TMP_DIR/BBnames.txt | grep -v "^$"| rev | cut -d: -f2- | rev | sort | uniq > $TMP_DIR/BBnames2.txt && mv $TMP_DIR/BBnames2.txt $TMP_DIR/BBnames.txt

cat $TMP_DIR/BBcalls.txt | grep -Ev "^[^,]*$|^([^,]*,){2,}[^,]*$"| sort | uniq > $TMP_DIR/BBcalls2.txt && mv $TMP_DIR/BBcalls2.txt $TMP_DIR/BBcalls.txt

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis

Expected Behavoirs:

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis perform well

Actual Behavoirs:

(aflgo) h3d@h3d-AS-4124GS-TNR:~/HUCHENG/project/targets/llvm-project/build$ $AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis
(0) Constructing CG for /home/h3d/HUCHENG/project/targets/llvm-project/build/bin/llvm-dis.0.0.preopt.bc..
(1) Computing distance for callgraph
(2) Computing distance for control-flow graphs (this might take a while)
awk: fatal: cannot open file `/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZN4llvm12function_refIFPNS_14GlobalVariableEvEE11callback_fnIZN12_GLOBAL__N_117DataFlowSanitizer7runImplERNS_6ModuleEE3.dot' for reading (No such file or directory)
**awk: fatal**: cannot open file `/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZZL28predictValueUseListOrderImplPKN4llvm5ValueEPKNS_8FunctionEjRKN12_GLOBAL__N_18OrderMapERSt6vectorINS_12UseListOrderESaISB_EEENK3.dot' for reading (**No such file or directory**)
**mv**: cannot stat '/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot': **File name too long**
@strongcourage
Copy link
Collaborator

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

@martinclauss
Copy link

Reading the error messages you get, the problem might just be the maximum allowed length of filenames. Check the limits with:

getconf -a | grep -iE '\b(name_max|path_max)\b'

on my machine:

NAME_MAX                           255
PATH_MAX                           4096

So in your concrete example:

echo -n 'cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot' | wc -c
257

As far as I know, there is no easy way to increase this limit since it depends on the Linux kernel and file system driver.
I don't know the details of how important this filename is for AFLGo but maybe it can be shortened somehow (e. g. hash of the mangled symbol?)

@DigOrDog
Copy link
Author

#138 (comment)
As mentioned above, I have indeed found that the issue is caused by the excessively long file name. It's easy to see that this file is generated by executing

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis.

So, I'd like to ask if there is a solution to this problem.

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

@DigOrDog
Copy link
Author

Reading the error messages you get, the problem might just be the maximum allowed length of filenames. Check the limits with:

getconf -a | grep -iE '\b(name_max|path_max)\b'

on my machine:

NAME_MAX                           255
PATH_MAX                           4096

So in your concrete example:

echo -n 'cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot' | wc -c
257

As far as I know, there is no easy way to increase this limit since it depends on the Linux kernel and file system driver. I don't know the details of how important this filename is for AFLGo but maybe it can be shortened somehow (e. g. hash of the mangled symbol?)

Thank you for pointing out the root of this issue. I also wonder if this file name will have any impact on the AFLGo process. Hashing does seem like a very good solution. I'm not sure if it's feasible.

@DigOrDog
Copy link
Author

DigOrDog commented Oct 7, 2023

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

Hi @strongcourage
Today I was able to fuzz LLVM with AFL, so using AFL to fuzz LLVM is not an issue. I'd like to use Hash with AFLGO to address the problem of long file names, but I'm not sure if this would affect the code logic. I hope you can provide a response, thank you.

@SonicStark SonicStark mentioned this issue Dec 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@strongcourage @martinclauss @DigOrDog