 sj |
||
|
||
Tech: Playing with the DT
|
|
|
Subject: Tech: Playing with the DT From: Jon Freeman Date: 12 Jul 22 - 09:01 AM A recent drift in a MIDI request thread and reading part an old dt thread refreshed by a spammer got me thinking about the dt database again. I thought I’d try something that would work locally with a browser without anything having to be installed or an Internet connection needed. I have used sql.js to handle the data which I’ve put in a sqlite database and (having failed with my attempts with abc2svg) abc.js to render the notation and play the tune (where one exists). It’s rather crude and needs more work that I’m unlikely to get round to doing but if anyone wanted a look at it as it is, it’s at http://www.jonbanjo.com/temp/dtbrowse.zip. It just needs all the files extracted to a folder and index.html opening in a browser (tried here are Chromium, Firefox and Opera on Linux). When the page id first opened, it will need you to choose the database. This is mysqlitetest in the folder the files were extracted to. I’ve not worked out how to just have the page load the database automatically – browser security gets in the way of that with a local file. Download is about 8MB but the dt database is quite large. |
|
Subject: RE: Tech: Playing with the DT From: Newport Boy Date: 12 Jul 22 - 11:52 AM That's great, Jon. Works well in Vivaldi on Linux. It's a faster way of looking through the DT. Thanks Phil |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 04:23 AM I’m glad it’s of use, Phil. As I indicted before, it wasn’t intended as a complete solution. It was more an experiment to see how an off line platform independent dt version might be achieved and it was intended to be hacked out fairly quickly. (If you can call taking getting towards two days on it quickly – It took me quite a while to work out how to use the 3rd party JavaScript code, and I had to modify my existing [I did have my conversion to a php/mysql version as a starting point] php attempt to convert sw to abc on the fly to fill a database column with abc ...). I guess an offline version is a bit late in the day for most anyway. Most people are connected wherever they go with phones and, on a different line of thought, I don’t suppose there will ever be another version of the dt released so that’s what I think was a historical problem gone. |
|
Subject: RE: Tech: Playing with the DT From: MaJoC the Filk Date: 13 Jul 22 - 05:58 AM .... I've been gently pondering the idea of (hm) a mirror site for the Cat, possibly (hm squared) a cacheing mirror in the first instance. Sadly I don't have the finance (or these days the knowledge) to do it myself, so I hereby launch the idea as bread on the waters. ducks and runs for cover |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 06:55 AM Hmmm. The Mudcat forum is huge (several GB of data) and I'm not sue how you'd get on with permissions but you could always ask Max. (While Mudcat is the main site for the DT Database, it's something created by the late Dick Greenhaus and Susan of DT and it used to be freely and widely circulated in various formats and others including myself have been able to make out own attempts from these). |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 07:01 AM hmm, I managed to paste some junk in my previous post! Hopefully a mod will tidy up for me. |
|
Subject: RE: Tech: Playing with the DT From: Stilly River Sage Date: 13 Jul 22 - 10:13 AM Did I trim it correctly? I enjoy playing with this kind of problem but right now I don't have the concentration (too hot and the knee surgery always knocks me for a loop attention-span-wise). |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 10:19 AM No, the url shouldn't be there. I'm not surprised you are struggling btw. The Texas heat alone would finish me! |
|
Subject: RE: Tech: Playing with the DT From: Stilly River Sage Date: 13 Jul 22 - 10:33 AM Ok, I trimmed the rest of the URL (it originally repeated twice). I usually leave my computer on over night in "sleep" mode, but for now I turn it off because I don't need the extra heat and because the power is liable to flicker if they do any brown-outs to spare the power grid. I used to have a download of the DT but it's probably three computers back and I don't think I moved it over. Is there still a downloadable old version? |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 10:54 AM Yes, the 2002 versions are at https://mudcat.org/download.cfm |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 02:47 PM I've had a different play today. At 25MB, I'm not sure anyone would want to download it but I've put it at http://www.jonbanjo.com/temp/DTDB.epub anyway. It's a first attempt at getting the dt into an epub ebook. |
|
Subject: RE: Tech: Playing with the DT From: Newport Boy Date: 13 Jul 22 - 04:37 PM That is a fascinating idea. I would be able to carry the DT around on my eBook reader. I'm downloading it to test tomorrow. (25MB is fine - I regularly download 1.5GB ISOs). Phil |
|
Subject: RE: Tech: Playing with the DT From: Joe Offer Date: 13 Jul 22 - 05:44 PM I have mixed feelings about messing with the Digital Tradition. When we couldn't agree about how to update the DT, we decided to leave it as it was at the last major update - 2002. We have been adding and correcting ever since, but all those additions and corrections are in the threads. For the most part, the additions are indexed under the following two indexes in the Quick Links dropdown menu: In case something happens to Mudcat. the threads and the DT are archived at the Library of Congress and at archive.org. |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 13 Jul 22 - 06:26 PM I'm not sure that's right Joe: The Mudcat Café TM There seems to be a bit of doubt on this but as far I can make out, Mudcat did get a 2007 upgrade but that version was never circulated in the other formats. I've never made any corrections or additions to anything I've done with the dt. I guess there is always the chance of something getting lost or going a bit awry with a conversion but, to the best of my ability, it's simply the 2002 (at least I'm pretty sure it's that) iversion in other formats. |
|
Subject: RE: Tech: Playing with the DT From: Joe Offer Date: 13 Jul 22 - 06:31 PM Thanks, Jon. I forgot about the 2007 update. It is only available here at Mudcat in online HTML, not in download form. There was a DOS version of the DT that came out more recently than 2002, but there were technical issues and the download version was changed back to 2002. |
|
Subject: RE: Tech: Playing with the DT From: GUEST,.gargoyle Date: 14 Jul 22 - 03:17 AM I have a set of five, dual sided 5.25 floppy disks from around 1998. At that time if you mailed the empty, preformatted, floppies to Dick Greenhouse with a SASE (self addressed stamp envelope) he returned them with the DT in plain text. It had to be a labor of love on Dick and Susan's part. Each floppy took perhaps a minute to load. I believe they were mailed to Palo Alto, CA. Sincerely, Gargoyle They reside in the right side middle drawer of a roll stop desk. I wonder if the mildew and silver fish have been kind? |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 04:35 AM I'll be interested to learn how it goes, Phil. One problem I have noticed concerns a Calibre conversion to mobi which seems to be needed for Kindles. The mobi winds up at over 200MB but maybe mobi is like that. But Calibre can't view its own conversion and I've not done well trying to find an Android app to work with it. It does work with dad's Kindle but the svg graphics are shrunk. The epub version seems ok for appearance using Lithium on my phone (at least allowing for my eyesight...) and on my tablet. I don't have an e reader but would like to try it on mum's Kobi. Unfortunately, she's mislaid that... |
|
Subject: RE: Tech: Playing with the DT From: GUEST Date: 14 Jul 22 - 04:54 AM I'm not sure when it happened but me and the English language seem to have gone our separate ways! |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 14 Jul 22 - 07:09 AM I tried the epub on my newish Android tablet with Bookreader It was e x t r e m e l y slow and only showed the text - I wondered if it had any tunes in it? Bookreader is pretty old and basic; I suspect it needs a newer app with better memory management. Here on desktop the default document reader, Xreader, opens it but fails if I click on a song in the TOC. I installed Calibre (which had gone missing at the last OS version upgrade). In Calibre I see it does have tunes and looks good - apart from a few extraneous entries in the TOC. I didn't dare download it to my basic Kindle Reader. It would convert to mobi, as you said, but the device often struggles with my newspaper. |
|
Subject: RE: Tech: Playing with the DT From: Newport Boy Date: 14 Jul 22 - 07:11 AM That works very well. I'm using a Sony eBook Reader, now 10 years old, and transfer was straightforward using Calibre. Download was quick - about 15 secs - but Calibre's preparation for first read took a while. About 8 minutes on my AMD Ryzen 5 5600G, pushing the CPU to 55%. Transfer to the Sony was quick, giving me a book of 13,400 pages. Selection via the alphabetical index is fine and the search function is effective but very slow - about 2 mins to find "four green". Not surprising with 13,400 pages! The sheet music images are a bit small but I would rarely want those. I will find this useful. As you say, most people will have access to the DT via their phones, but I don't use the browser on my phone. The place away from home where I'm most likely to want to refer to the DT is our caravan, where the mobile phone signal is intermittent at best - texts generally get through in half an hour or so and I usually know there's a voice call and can go up the dune to respond. I can't help with Kindle. I rejected it 10 years ago - I don't like locked-in proprietary hardware. Phil |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 07:31 AM Thanks for the feedback. I'll see if I can make the images a bit bigger if/when I do an update. The first reader in this household was a Sony I chose for dad. My reasoning at the time was the same as yours. When that broke and he wanted a new one, I went with Kindle thinking it might be nice for him to be able to use it with his Amazon account. That didn't work out as the damn thing churns out Prime Book Recommendations or something like that and I found that he was accidentally buying e-books he didn't want. End result is I set a parental lock on the device to stop him doing that... |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 14 Jul 22 - 07:35 AM Joe Offer wrote: the threads and the DT are archived at the Library of Congress and at archive.orgI didn't know about the Library of Congress web archive. But it appears to have stopped in May2021 https://webarchive.loc.gov/all/*/https://mudcat.org/ |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 07:49 AM I'd missed Dave's reply. I know about the extra lines in the contents page but don't understand where they came from. They are not in the index.html file I passed to Calibre for the conversion to epub. I'm not sure if you know but the epub is just a collection of html and svg files in an archive with some other data added. It will open with ark and other archive software. |
|
Subject: RE: Tech: Playing with the DT From: Richard Mellish Date: 14 Jul 22 - 08:01 AM Jon's link takes me to the download page https://mudcat.org/download.cfm OK, but attempts to download Windows Version: tdtsetup.exe (7.66 Mb) from there are failing. |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 08:27 AM The other files there start downloading for me but tdtsetup.exe (7.66 Mb) is just hanging. Maybe Joe can help. I don't know of an alternative download source for this. |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 14 Jul 22 - 10:48 AM I can download tdtsetup.exe from the archive here https://web.archive.org/web/20101126034502/http://mudcat.org/download.cfm I wouldn't expect it to work on a recent version of Windows, though. |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 14 Jul 22 - 12:55 PM Jon Freeman wrote: The mobi winds up at over 200MB ...My Calibre (v6.0) converted the 26MB epub to a mobi of 535MB. And does open it. Perhaps yours was incomplete? Other ordinary books I have are only slightly bigger in mobi format. Jon Freeman wrote: ...epub is just a collection of html and svg files in an archive...Unpacking it, I see that the the SVG files (4329 of them) are 77MB and the rest (9011) 47MB. I wonder if the mobi has converted the compressed SVGs to some other image format, each at several sizes for different Kindles? The sqlite database of DT is 24MB. Poking about in it with sqlitebrowser I can see is contains lyrics in html (8982 records), tunes is ABC and SongWrite (4020), and keywords (711). (Mudcat's version appears to have 10365 songs.) Your prototype webapp displays the lyrics in staff notation and can play midi - both from the ABC presumably. It could display, and offer to download, the ABC and the Songwrite source. Made into a proper webapp, loadable from the web but with the ability the run offline, it would be pretty neat. I estimate that the posts in the Mudcat forum occupy about 5TB. That's not including other entities - threads, people, ... or indexes. Perhaps 10TB in all? |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 02:01 PM I used Calibre 6.0 too. Dad's Kindle does work with the mobi so I guess it is all there although I've struggled with anything else I've tried... If you start a conversion manually, Calibre does offer a range of input and output profiles so it's possible the program picks the right profiles for the Kindle device it. It's also possible that mobi needs a different graphic format to svg. I don't know. One thing I could easily try is to see what happens with png graphics. The SQLite database contains more that it needs at the moment. I'm not sure there's any point in having the SW there at all. I could make it downloadable but it's an obsolete proprietary format. Keywords aren't used but I'm unsure whether I'd want them or not if I did have another play with the app. In the thread I quoted Joe on about release versions, Susan of DT says: There are 10,000 songs in the Feb07 edition, vs. 9,000 in the 2002 edition, roughly.<\blockquote>So I think your song counts are about right. |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 14 Jul 22 - 02:59 PM I've just tried a png version. The epub conversion has gone up to 57MB the mobi conversion (which again Calibre can't read) came out at 197MB. One thing I've just noticed is that the svg mobi produced automatically for dad's Kindle was 140MB - the 200+ size I quoted before was for a (now deleted) manual.conversion. I think there must be some resizing of graphics depending on profile/devices. |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 14 Jul 22 - 03:28 PM If I download to my kindle via USB it offers to convert (and doesn't download if I decline); I didn't do that, as I said. I did a 'manual' convert. It didn't ask what device. It took ages - half an hour? Newport Boy wrote: The sheet music images are a bit small but I would rarely want those.When I'm waiting my turn to sing I need the music. Otherwise I forget how the tune goes! My tablet has a zip file with dozens of pdfs, each with the staff notation (often transposed) and the words. |
|
Subject: RE: Tech: Playing with the DT From: Joe Offer Date: 14 Jul 22 - 03:40 PM Here's a link to the Windows version: http://joe-offer.com/DTWindows/DigitalTraditionWindows.zip |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 15 Jul 22 - 12:10 PM For the epub, I’ve reduced the pagewidth for abcmps in the graphic conversions. This should make the graphics bigger but they may need to show more lines of staff than before. I’ve also reduced the size of the title font. I’ve pit a small sample (just the first 100 songs) at https://www.jonbanjo.com/temp/dttest1.epub I’d be interested to know if others find this an improvement. |
|
Subject: RE: Tech: Playing with the DT From: DaveRo Date: 15 Jul 22 - 01:09 PM I installed that smaller version on my basic Kindle. It auto-converted the 300KB epub to 6MB - which is the same inflation factor as the full version. It works OK. But a 6" Kindle is really too small - my tablet is 10". The staff notation is readable, though - I could play that at a pinch. The old Digitrad Joe linked to works OK on my wife's Win 10 laptop. I put it on a stick and ran it from there. I like that the songs with tunes are in bold on the index. Feature suggestion for any future webapp version: ability to transpose the pitch of the displayed music. That would be something the original can't do! |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 15 Jul 22 - 04:15 PM Thanks Dave. I've had the DT Windows running under wine. No sound but that may well be down to my MIDI configuration. I'll add bold for songs with tunes to the epub and (if do more with it) the app. abc.js supports both score and tune transpose (they work independently) so that could be added to the app. |
|
Subject: RE: Tech: Playing with the DT From: Newport Boy Date: 16 Jul 22 - 07:17 AM Jon Freeman wrote: I’ve pit a small sample (just the first 100 songs) at https://www.jonbanjo.com/temp/dttest1.epub I’d be interested to know if others find this an improvement. That's a great improvement here. The staff notation is now usable, rather than barely readable. Phil |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 16 Jul 22 - 07:26 AM Thanks Phil, I'll leave things a little while in case anything else is suggested but I'll aim to put a new complete version up sometime next week. |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 20 Jul 22 - 06:02 AM I've put a new version of the epub at https://www.jonbanjo.com/dt/epub/DTDB02V2.epub This has the larger graphics and songs with tunes in bold. |
|
Subject: RE: Tech: Playing with the DT From: Newport Boy Date: 22 Jul 22 - 05:38 AM Thanks, Jon. That's on my Sony Reader now and all very readable. Just for interest, Calibre took about 3.5 minutes to format for first reading. This time it showed the CPU at 72% and pushed all 12 cores to maximum speed. I don't usually see anything above 25%. Phil |
|
Subject: RE: Tech: Playing with the DT From: Jon Freeman Date: 22 Jul 22 - 07:20 AM I don’t know about cpu usage Phil but some of my conversions and first reads have taken a bit of time. I’ve nothing as fast as yours but my desktop PC (on which I run the Calibre stuff) is a Ryzen 5 3400G and my “server” PC (on which I make the html pages for Calibre to convert – I make use old folkinfo code for some of this) is a Ryzen 5 5600X. I think the longest job I had was one on the server where I set ghostscript to 300dpi for ps2png conversions (abcm2ps does svg and postscript output but gs gets added to the chain for png). I guess that job took about 45 minutes for the full set of html pages and graphics. The result (at: https://www.jonbanjo.com/dt/epub/DTDB02V2png.epub) doesn’t look too bad on mum’s (now found..) Kobo but at 170MB, I’d not recommend it for general use. And certainly avoid it for epub to mobi conversions (my try came out over 1GB). It can however be tried if a device using epub is having problems with fonts and svg graphics. At least until I find a better solution _ I’ve tried a few ideas on this but all so far have failed. |
| Share Thread: |