Group 19 Discussion

[laurendubois]

A Group 19 discussion board with new topics in each thread below

[lpdbmid]

Meeting One Notes

[lpdbmid]

Agreed to share scripts we are working on and perhaps work through them together to see how they work
Agreed to share list of functions / tasks we want to achieve with our python scripts

[lpdbmid]

Use discussion board for meeting notes

[lpdbmid]

For next meeting
Go through both provided how to scripts

[lpdbmid]

Python courses
https://automatetheboringstuff.com/

[lpdbmid]

https://www.coursera.org/learn/python/home/week/2

[lpdbmid]

Here is the instructor page (Python for Everybody) that is not tied to Coursera but has all of the same content, books/lessons
https://www.py4e.com/

[laurendubois]

Pycharm tutorial/setup - https://www.youtube.com/watch?v=56bPIGf4us0
Python for Beginners - Learn Python in 1 Hour - https://www.youtube.com/watch?v=kqtD5dpn9C8

[laurendubois]

Introduction thread - could we comment here with our name/role/organization, version of python and IDE, anything else?

[JoCastle]

Jo Castle / Photographer / University of Manchester : John Rylands Research Institute & Library Imaging Team
Python 3.12 on Windows using VSCode

[lpdbmid]

Laura Peaurt, Archivist (Digital Preservation) The University of Nottingham, Python 3.11, IDLE GUI on Windows

[lwforpres]

Lauren Work, Digital Preservation Librarian Yale Library, Python 3.9.6, Sublime text editor, checking out other IDE options

[laurendubois]

Lauren DuBois, Metadata Analyst & Librarian, Fondren Library at Rice University

Python 3.8. Pycharm on Windows

[CRKussmann]

Carol Kussmann, Digital Preservation Analyst, University of Minnesota.
Python 3.12, not sure which IDE I will be using yet.

[JoCastle]

Existing scripts: A place to list and describe any scripts we have already written

[JoCastle]

Hi @laurendubois, sorry it's taken me a while to reply to this. In answer to your questions:
A) I included a font parameter as a design choice, I was trying to make it look nice but it's not necessary for it to run
B) I cannot envision needing to run this on the same zip file twice. Once it unzips the files and makes an appropriately named folder to put them in there's no reason for our team to do that again, as we now have access to all the files. However I do think this is probably a case where best practice would say I should include error-handling to say why it's not working in case a use does try to do this.
C) I also get a permissions error- "PermissionError: [WinError 5] Access is denied"

***Running the code I sent to check the exact permission error I get has shown me that there is a problem with this code and it does not work properly. It doesn't drill down into all of the sub-folders for some reason, the code doesn't seem to be there to do that. I'm not sure why, I have a version packaged into a .exe file which I have distributed to my team and that version is working, so I must have changed something somewhere to break it, I'll got back to my git versions, try and see what I've done and repost the working code.

Regarding sharing a zip file containing TIFs, I checked with my manager and I can do this, as long as everybody agrees to use the tifs inside only for testing this code, as they are images of our collection and I don't have authorisation to distribute them for any other use. I'll upload them once I get a working version of the code.

[JoCastle]

Hi @laurendubois sorry it's taken me a while to get back to you on this. I've had students in doing placements for the past couple of weeks and they've basically taken all of my time.

I have fixed the PreservicaUnzip script, and I tried to upload it to the folder that was created for our scripts but couldn't. I've submitted a question asking for help in the Discussions but here is the fixed script as a text file in the meantime:
PreservicaUnzip.txt

Once I can upload files to the shared area I'll also upload a sample Preservica download for you to test it on.

[JoCastle]

Also, I have just got it to delete the two directories and all sub-directories and files it creates. That's not in the script I've attached above, as I literally just found the article that told me how to do it.

I was trying to use os.remove, which it turns out is just for files not directories.

os.remove(save_path+'/DELETE ME')  #results in permissions error, does not delete dir
os.remove(save_path+'/DELETE ME2')  #results in permissions error, does not delete dir

To delete non-empty directories I have now discovered there's a function that's part of the shutil module, which is a module I'm already using in the script to move files around.

shutil.rmtree(save_path+'/DELETE ME')
shutil.rmtree(save_path+'/DELETE ME2')

I have tested this in my script and got it to work, here is the article I got the info from.

[lpdbmid]

Hi - I have now managed to test this script and spend a bit of time making sure I understand what it is doing. So helpful thanks!

I wonder if anyone knows how to make a script keep checking lower and lower in a folder hierarchy for nested zips? We have some hierarchies that go a few levels of nested zips, and it seems like standard zip extraction only goes a certain number of levels down?

[CRKussmann]

Not a python script, but I've used this in the Windows command line to just get a file list of a folder/nested folders as a txt file. Navigate to the folder you want to create a file list for, and this will create a txt file in that folder called filelist.txt.
Create file list on Windows with: dir /b>filelist.txt.
Recursive list: dir /A-D /S /B >filelist.txt

[JoCastle]

Desired outcomes: Problems we would like to solve / Tasks we would like to automate using Python

[JoCastle]

Work with the Adobe PDFServices API

[laurendubois]

Change the metadata in a set of PDFs

[laurendubois]

Ultimate goal - With Python, use Alma's API to retrieve metadata, utilize WorldCat's API to compare the extracted metadata with our holdings in WorldCat, then update the holdings in WorldCat if discrepancies are found.

[laurendubois]

Find all instances of "phrase" and replace with "newphrase" across several thousand bibliographic records

[CRKussmann]

Compare the checksum values (spreadsheet/csv) between two files. (Not comparing rows, as they wouldn't be in the same rows). Needing to compare files with 50K+ lines. Looking for unique values and then outputting the unique values only into a file.

[digitensions]

Hi group 19! Wow you guys are super busy. Just wanted to say hi! Happy to give some python pointers if needed/I can this month. Joanna

[lars-unimelb]

Hi, mentor here. One of the things that have happened since that stack overflow post is the new pathlib library that can simplify that process.

For example, this would do what you're describing:

import pathlib

root_dir = '/Users/lauren/Desktop/LW_SIPCreatorCLI'

for metadata_xml_file in pathlib.Path(root_dir).rglob('*/PROCESSED/metadata.xml'):
    print(metadata_xml_file)

You can read more on pathlib and what you can do with pathlib file objects here: https://realpython.com/python-pathlib/

[lwforpres]

Thanks so much! I tested out pathlib and your suggested script and it works just as I had hoped. Thanks very much for this resource and for pointing me in the right direction.

[lars-unimelb]

No worries at all :)

[CRKussmann]

Going back up to the first question I posed (sorry I can't see where/how to put it closer to my question)... but the suggestion was to use dictionaries and linked to this help There's good advice for comparing key/value dictionary pairs in lists here:
https://stackoverflow.com/questions/9845369/comparing-2-lists-consisting-of-dictionaries-with-unique-keys-in-python
Can someone confirm what the code on this page is saying. It is saying that it is comparing the PAIRS (both columns)? I only want to compare one column (and not row by row but overall) but output two... Just looking to see if this is what I should be looking at. Thanks.

[laurendubois]

@CRKussmann I believe the dictionary would pair up the checksum values in the first file and the checksum values in the second, so that would be the "pairs." Then you would write code to see if they do/don't match. Or thats how my recent dictionary worked, anyway

[laurendubois]

Hello! Just wanted to post I am making progress on one of my "Desired outcomes/Problems to solve." Yesterday I worked with my supervisor to gameplan how to check our holdings with OCLC using the WorldCat Metadata API. Basically, the problem is we have some items in Alma that are linked to an OCLC# in the record, but, for some reason, WorldCat doesn't reflect that our institution holds that item.

We put a list of OCLC numbers into a Google Sheet and then ran the script - it would use the [WorldCat current holdings API] to check the OCLC ID against the WorldCat holdings, and then spit out True/False if it shows our institution holds an item associated with that OCLC ID. I can share the basic code here, but I think I should wait because we are still solving problems like -

1. Where should the output go? Right now, it just says OCLC ID = true/false in a list in the terminal. I think we want it to write true/false back to the Google Sheet so we'll have a list of records to update.
2. Do we want to document records that were reviewed, only the records that were changed, or something else? Where/how?
3. Ideally, we would expand the code to create an if-then statement, like "If we have this item but WorldCat says False, use [WorldCat set holdings API] to change it to true."
4. How to handle errors. Right now, we can run it on a long list, but if one of the OCLC IDs in the list is no longer in WorldCat or there is a blank, the script breaks/stops running. We need to add steps on what to do when it runs into this issue so that it keeps working through the end of the list.

Fun!
LD

[lwforpres]

Hi all,
Adding a reminder here today that we are meeting again in a week's time.

We have all been working hard over the last few weeks and I'm excited to hear more updates from the work we have been posting about here. As a reminder, we had generally agreed to use the next meeting to:

• Go over both "how-to" provided scripts (There are now 3, which is exciting, but I think the originals included the batch rename and removing unwanted files, but we can cover all if needed)
• Talk about the scripts and work (and progress!) we have been sharing here

[laurendubois]

Just a funny exchange with my supervisor on Slack I thought you all might appreciate! :)

[lpdbmid]

Apologies - zoom installing new update!

[laurendubois]

Hello - thanks for the great chat today! Here is the code that looks at two XLSX files, and creates a third XLSX file that gives results of whether the OCLC ID is found (in both files) or missing (in the OCLCIDs1 file). Maybe OCLCID columns could be replaced with the checksum values?

[laurendubois]

Hmm I was going to paste in here but can't get it right. Attached instead!
Compare-Two-Lists_forPSG.txt

[laurendubois]

Update - we ended up going with a modified version of this --> https://github.com/colinbitter/OCLC-simple-reclamation

[laurendubois]

FYI - here is the 100 Days of Code: The Complete Python Pro Bootcamp with Udemy. I'm on Day 5 and already learned so much. Right now it's $18.99 for two more days, and you have lifetime access (so you don't have to start it right away. I think I sat on my signup for over a year LOL.)

[(https://www.udemy.com/course/100-days-of-code/?couponCode=ST15MT31224)]

[laurendubois]

Here to add I finished Day 5 and made a fun treasure island game. You can see my version on Replit (free web-based site), and "fork and run" or paste it into your IDE if you want to play it yourself (try not to look at the spoilers!)

https://replit.com/@laurenmd/treasure-island-start#main.py

[lpdbmid]

Hi all - I'm really sorry but I don't think I am going to be able to make the meeting today as I have had something put in my calendar. I will see if I can have it moved.
Apologies again

[laurendubois]

No worries!

PS - is anyone else able to join? I've been sitting in the waiting room for ages. It just says "the meeting host will let you in soon." Perhaps I've got the wrong number?

[lpdbmid]

Is it because of shift to summer time or something? Perhaps it doesn't start for another hour? (I have it in my calendar as not starting for another 50mins)

[laurendubois]

Yes! I just came back to say this - I forgot about the time switch! I'm an hour early. Unfortunately I won't be able to make it either then, as I've got a conflict at 9 my time. So sorry!

Here is a new tool I've used this week - Google Colab - to share my treasure island game. Again it's web-based, so you can just go to this link and click the triangle "run" symbol to play it :) And see my code too! https://colab.research.google.com/drive/1-n0CCSdbjPqpLR7CzsKra0BOJewZn3pI?usp=sharing

[lwforpres]

I also forgot about the time switch, and sadly have a conflict at as well. Hoping to get a bit more done before our next session to share! @laurendubois thanks for sharing your game and Google Colab!

[laurendubois]

Hi there! Excited to join our next meeting now that I have the times corrected :)

Today one of our digital scholarship/preservation colleagues asked about using Python scripting to crop digital images. Like not just change it to a specific size, but do all the finding the focal point and fixing the skew and all that too. Anyone doing anything like that?

Thanks,
LD

[lpdbmid]

Hi there - I haven't done this but would be interested - does googling throw up any ideas? I have been regoing through the python for everybody course (which I had previously started) to try and cement in some facts! - Wondered if anyone else has been doing this course recently? I know somebody recommended it during the first meeting.
I have added another task I want to look into which is to create a python script to run DROID from the command line - select a folder to analyse and save the output into a specific folder for use in another script at a later time. Does anyone have any familiarity with using DROID in this way rather than opening up the GUI? I'll do a bit of exploring anyway!
One more thing - I've had a go at using the weather API example and it has 'worked' but rather embarassingly I'm not sure I still really understand how - I realise this script may have already been discussed in the meeting I missed!

[lwforpres]

@laurendubois That is a really interesting question that I think it's likely there may be some libraries for - you might have already found PIL (now forked and maintained as Pillow) that looks like an interesting.

@lpdbmid I have been (slowly and inconsistently) going through python for everybody and plan to get back to it! I'd be curious of your experience so far. Your task for DROID sounds like a really useful one - in prior roles I mostly ran it through the GUI but was interested in using the XML outputs for use in scripting.

Looks like we are set to meet at a slightly updated time for all - I can only attend 10-11am (East Coast time, 14:00-15:00 UTC) but look forward to catching up

[laurendubois]

WEATHER API THINGS -

Here is a Github link to my code - https://github.com/laurendubois/ForFunsies/blob/main/weather_API.py

Here is a pic of what it looks like in Pycharm vs. regular NotePad & Github -- all the default colors to separate variables from text from definitions from notes, etc. are so helpful!

[laurendubois]

Running this results in:

[laurendubois]

Sorry I missed you all last month! A catastrophic storm hit my house/cars. Looking forward to seeing you for our final session tomorrow!

Lauren

[lpdbmid]

Sorry to hear this, I hope you are ok. I will be attending this afternoon
I'm a bit disappointed that I have not been able to give the time I want to furthering the early work I did the last two months :-( However I have been prompted this week to catch up with continuing on through my python course!
Everything you guys have shared has been really helpful though!

[CRKussmann]

Really hoping to join! In Oregon teaching a course and might be on the move and not able to be on the computer. Anyone continuing? - Carol

…

On Tue, Jun 25, 2024 at 6:34 AM Lauren D ***@***.***> wrote: Sorry I missed you all last month! A catastrophic storm hit my house/cars. Looking forward to seeing you for our final session tomorrow! Lauren — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF7MXQXF6F26RRUTI5LJB2TZJFWV5AVCNFSM6AAAAABCI3OPEOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQNZRHA2DI> . You are receiving this because you were mentioned.Message ID: <Digital-Preservation-Coalition/PythonStudyGroups/repo-discussions/9/comments/9871844 @github.com>

[lwforpres]

I should be able to make the first hour today! I did sign up to continue - even though I haven't gotten nearly as much done as I hoped, having the group is still a nice way to learn and ask questions, as well as accomplish a few small things.

[lpdbmid]

I have just asked about continuing although I can see I missed the deadline so will have to see! (Also I got my time zone stuff wrong - again and was trying to join now! See you all in a little while...

[mpstrickland]

[like] Manuela Pallotto Strickland reacted to your message:

…

________________________________ From: lpdbmid ***@***.***> Sent: Wednesday, June 26, 2024 1:09:27 PM To: Digital-Preservation-Coalition/PythonStudyGroups ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Digital-Preservation-Coalition/PythonStudyGroups] Group 19 Discussion (Discussion #9) I have just asked about continuing although I can see I missed the deadline so will have to see! (Also I got my time zone stuff wrong - again and was trying to join now! See you all in a little while... — Reply to this email directly, view it on GitHub<#9 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOU3S2VTT57DRBNY5XOH72DZJK4QPAVCNFSM6AAAAABCI3OPEOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQOBSG4YDA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>