Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
4th August 2017, 14:46 | #1 | Link |
Registered User
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
|
How to make StaxRip download easier for people with low bandwidth?
Is there a simple way I can use to make it easier for people with low bandwidth to update to the latest test build?
Right now StaxRip includes 82 tools and is 97 MB packed, about 400 MB unpacked.
__________________
https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists Last edited by stax76; 4th August 2017 at 14:49. |
4th August 2017, 15:07 | #2 | Link |
Registered User
Join Date: Dec 2002
Posts: 5,565
|
- slim packages down (e.g. only compile ffmpeg with the libraries used by StaxRip)
- divide package into small ones for every tool and only update those that have changed and only those the user actually uses - implement some diff update (e.g. based on xdelta3 - probably some tools out there) Xdelta3 test: Patch from StaxRip-x64-1.6.0.1-test.7z to StaxRip-x64-1.6.0.2-test.7z is only 827 kB. |
4th August 2017, 17:24 | #3 | Link |
Registered User
Join Date: Dec 2008
Posts: 589
|
Put the files in an archive/package file you can easily control/manipulate .. for example TAR files (512 byte header for each file/folder, then file content with padding to 512 bytes chunk) ... when you release an update you can make new tar while keeping the file names and order (in tar file) the same to make the diff (with tool like xdelta3) easier...
Download diff, apply diff to old tar file, delete previous version of the tar file. Optionally, compress the diff with 7zip or some other open source compression scheme. + agree with slim version of ffmpeg stay away from compressing binaries with tools like upx , too much hassle with antivirus programs to be worth it and would just lower your diff efficiency |
5th August 2017, 10:58 | #4 | Link |
Registered User
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
|
Xdelta3 would be a option, the initial tar would be about 400 MB though. Custom builds would be too much work, so far I only make a custom installer for avisynth and sometimes build mp4box (there is Visual Studio project in the repo which builds out of the box). At the moment it's already difficult constantly updating everything and answer the requests which lately are many. Another option would be uploading a light version excluding VapourSynth plugins, mpv and the hardware encoders and other things to cut the size minimum in half, this would only be available for test builds and people could overwrite the full version letting windows file explorer integrate it.
__________________
https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists |
5th August 2017, 13:41 | #5 | Link |
Registered User
Join Date: Dec 2008
Posts: 589
|
You shouldn't really care so much about how big files are, in the age of TB sized hard drives and games that use 60-80 GB.
People only care about how big an application is, if they have small capacity SSD drives as their boot drives. You have to AUTOMATE things. For example, every time your application changes either due to new versions of your tools or the main application itself, it gets a new release version, a unique ID. Let's say 2017080501 (for 2017 08 05 1st release this day). If tomorrow you update x265 in the tools folder, you'll have version 2017080601. Now you can write something that will automatically create those "update packages" ... the tool can build the 7z file like you have it on your website, but can also go back and create DIFFs for let's say the last 10 versions of updates... diff from 2017080501 to 2017080601 and so on... You can then write a small update tool and bundle it in the package. The tool can query a website and download the latest update information, which could be something like (of course encoded in a better way like using json or xml) : update version=2017080501 type=full size=400MB url=xyz sha256= update version=2017080501 type=partial version_previous=2017080401 size=10MB url=xyz update version=2017080501 type=partial version_previous=2017080301 size=15MB url=xyz So your update tool can then either download the diff from a version to the next one, or choose to download the full package. If applying diffs, your tool most likely will need to hash check each file that would be changed by the diff to make sure it's the proper version. If any of the files is modified the diff can't be applied so the update tool will have to give up and download the full package. You could look at torrent files to inspire from ... they're basically a collection of file paths along with several sha256 hashes for each file (one sha256 hash for each chunk of data in the file, where the chunk could be from 4K to 16 MB in size) |
Thread Tools | Search this Thread |
Display Modes | |
|
|