Quantcast
Channel: Planet Python
Viewing all 22875 articles
Browse latest View live

PSF GSoC students blogs: Week #9

$
0
0

I explored different ways in which PCA can be tested. More specifically, I tried to find some testing schemes under which the PCA I developed fails while the standard PCA method works. This practice is to help me understand the potential limitations of the PCA method. Unfortunately, I have yet to find a case where the PCA method fails. So far, it appears that the standard PCA and the implemented PCA performs more or less on the same level. Currently, I'm trying to exploit the fact that "hyperbox" method was used by drawing the loading matrix from different heavy tail distributions.

What did I do this week?

Design test cases for PCA

What will I work on next week?

Devising test cases for PCA. Implement NNMF


PSF GSoC students blogs: Week #10

$
0
0

I continued working on developing test cases. I tried to write up an overview for testing frameworks since such a framework could potentially be useful for other methods beyond PCA. I still haven't found a test where the implemented PCA falls short of the standard PCA, which is good in the sense that the performance is on par with the standard PCA but potentially harmful since we don't know its vulnerabilities. Meanwhile, I'm in the process of implementing NMF. I'm currently am trying to use the hyperbox method that I used in PCA. Unlike PCA, however, it is generally not possible to apply NMF in an incremental manner as we did in PCA so I'm reading up papers to resolve this issue.

What did I do this week?

Design test cases for PCA. Partially implemented code for NNMF. 

What will I work on next week?

Testing framework for PCA and other methods. Continue implementing NNMF

Mike Driscoll: PyDev of the Week: Eric Matthes

$
0
0

This week we welcome Eric Matthes (@ehmatthes) as our PyDev of the Week! Eric is the author of the popular book, Python Crash Course. He also created a neat set of Python Flash Cards that I reviewed earlier this year. You can catch up with Eric on his website or check out some of his work on Github.

Let’s take a few moments to get to know Eric better!

Can you tell us a little about yourself (hobbies, education, etc):

Sure! I grew up in southern New Hampshire, on the outskirts of Boston in the early 1980s. My father was a software engineer at DEC around that time, and I first learned to program on a kit computer in our basement back then. I am so grateful to my father for sharing the technology he had at home, instead of telling me to keep away from it all. It has been amazing to watch computers evolve from the early days of almost no one having a home computer to almost everyone having multiple computers in their lives.

I loved math and science in high school, and I went into undergrad as a chemical engineering major because I loved AP Chemistry. But I soon found that engineering was really about learning to solve other people’s problems. I enjoyed my physics classes though, because they were all about understanding the universe, from the very large to the very small. For a while I naively worried that if I stayed with physics long enough I’d start to find the world less interesting as I understood it on a deeper level. It was a joy to discover that the opposite was true: the more I learned, the more fascinating everything around me became.

I continued to learn new programming languages throughout my educational experiences. I took a variety of programming classes, and always had a few projects going for fun. I wrote a 3d graphing program in C during spring break one year in college.

I wanted to be a particle physicist, but I didn’t want to be a student forever. I decided to try teaching for a couple years, and quickly found that the intellectual challenge of trying to reach every student in my classes was just as satisfying as doing hard science. I loved teaching, and decided to stay with it.

In 2011 my son was born, and a month later my father died. It was a really hard time, but it was also a formative experience for me. My mother asked me to look through my father’s computer and let her know if there was anything worth saving. It was a really intimate experience, looking through all the projects he was working on, and reading through his notes. I used to visit him in his office whenever I went home, and as long as his computer was open and running that day I still felt directly connected to him. It was sad to realize these projects would never be finished, and would never be used by anyone. In the weeks that followed I realized that if I died you’d find a bunch of half-finished projects on my computer as well. I made a commitment to start using the skills I’d learned to build something meaningful.

I wanted to build tools that would bring greater equity to public education. I gave a talk at PyCon 2013 about how much the educational world could gain from the open source model, and Bill Pollock of No Starch Press approached me afterwards. “I hope you build what you described, and if you ever want to write a technical book let me know.” I went back to my classroom and saw a poster hanging on my wall: “What’s the least you need to know about programming in order to start building meaningful projects?” It was a list I had made for my students of the smallest set of things they needed to know in order to be able to build the things they cared about like games, data visualizations, and simple web apps. I realized that was the book I wanted to write, and the question on that poster became the guiding question for Python Crash Course. I hadn’t intended to write a book, but I realized that in five years of trying to teach programming to high school students, all the resources I found were either aimed at young kids, or assumed more technical knowledge and experience than my students had. I decided to write a book for anyone old enough to not want a kids book. It has been immensely satisfying to see that Python Crash Course works for almost everyone in that anticipated audience: young kids motivated enough to want a more serious book, high school students, undergrads in all majors, grad students, working adults, and retired people who are curious to learn programming at an older age. I was surprised to find it even works well for people who are already fluent in another language, and want to pick up Python quickly.

I don’t just do technical work. After growing up in New Hampshire, I moved to New York City for seven years in the 1990s-early 2000s. I bicycled across the US a couple times during the summer, and then lived on a bicycle for over a year at one point. I rode from Seattle to Maine, down to Florida, over to California, and up to Alaska. That was a life-changing experience, and I moved to southeast Alaska for good in 2002. I love living in a sizeable but isolated town right on the edge of true wilderness. When I go trail running, there’s a very real chance I’ll see a brown bear. Server crashes are a lot easier to keep in perspective when I’ve had close encounters with bears in the woods. I love steep mountains, and I’ve been an active member of a technical rescue team since I moved here. I also love living a partially subsistence life style. Every year we catch our own king salmon, sockeye, coho, halibut, crab, and deer. It’s wonderful living in a community where almost everyone goes out to fish, hunt, and gather a good deal of their own food.

Why did you start using Python?

My language of choice in the mid-2000s was Java. I had used C for a while, and liked the higher level libraries that Java offered. Then a friend said, “Hey you should check out Python. Your programs will be about a third as long as they are in Java.” Like many people new to Python at the time, I was pretty skeptical of the use of whitespace instead of braces, and I was surprised I didn’t need to declare variables. But these ideas were more intriguing than off-putting. I was working on a project to make artistic representations of random walks, and I had built a desktop application for generating and styling random walks. I rewrote the application in Python, and it really was about a third as long as my Java program. I was floored, and I’ve been using Python for almost every project since then.

When asked why they use Python, people famously say, “I came for the language and stayed for the community.” That answer certainly speaks for me. After my father died and I decided to take my programming work more seriously, I looked for a technical conference to attend. I am so grateful that I stumbled upon PyCon. I first went in 2012, which happened to coincide with some of the early diversity and Code of Conduct initiatives. I watched as the community set clear diversity goals, made specific plans to achieve those goals, and followed through on those plans. When I go to PyCon now, I see a fully diverse crowd, and the variety of topics that are presented is just awesome. The Python community is answering the question, “If we make programming accessible to everyone, what will people build?” It’s critical that we give people from all walks the power that programming offers, and let people build projects that serve their own needs, and the needs of their communities.

What other programming languages do you know and which is your favorite?

I started with Basic in the 1970s and early 1980s. I never got too deep into any language until Java and Python in the mid 2000s, but along the way I gained some understanding of Logo, C, Pascal, Fortran, HTML, CSS, JavaScript, and SQL.

I use Python almost every day now. In the next year or so I really want to solidify my understanding of SQL, and I’d like to explore a functional language like Lisp or Haskell. I’d also like to play around with a variety of languages I’ve heard about but never used such as Rust and Go. I don’t ever want to get stuck with Python because it’s what I know; if something better comes along, I want to evolve with the industry.

What projects are you working on now?

I just retired from teaching to focus full time on writing and programming. As Python Crash Course became more established, supporting the book gradually became a second job. As a full time teacher with a young son, all of my writing work needed to happen before 7am or after 9pm. That’s been hard. I am so happy to be finished with classroom teaching, and be free to follow up on the many opportunities that have come up as a result of the success of this book.

My most recent project involves a local nonprofit that’s working to minimize conflict between longline fishing crews and sperm whales. Longliners set mile-long lines on the bottom of the ocean, with hooks every 10 or 15 feet. They target bottom feeders like halibut and sablefish (black cod). They let the line soak for 6 hours or so, and then haul the line back to the surface. Sperm whales have learned to hang out near longliners, and they can pick off almost every fish on a mile-long longline just before it’s brought to the surface. I’m working with a team that has fishing boats tow a hydrophone array on their way to the fishing grounds. The data from the hydrophone array is used to map the location of any sperm whales in the area, and these locations are broadcast to any boats in the area. The goal is to help everyone avoid fishing where the whales are, to the benefit of the fishing crews and the whales. (Longliners are *not* trawlers. Bottom trawlers drag nets across the bottom and destroy everything in their path. Longlining is a sustainable fishery, and has comparatively low bycatch rates.) My role in the project is to help get data from the satellite devices on board the fishing boats to the people who can map this data and get it back out to the boats on the fishing grounds. It’s a privilege to work with such an interesting group of people from the fishing and scientific communities.

I am also working on:

  • Updating the Cheat Sheets that accompany Python Crash Course. These have been immensely popular, and I look forward to releasing the updated version later this summer.
  • Adding new sections to the online resources for Python Crash Course. I’m planning to add a set of Challenges that go beyond the exercises in the book. I also want to write a series of articles and discussions that build on what people have learned in the book. I’m planning to develop a set of teaching guides as well, to support the use of the book in academic settings.
  • I’m working on analyzing local weather patterns to help address the impact of climate change on our community.
  • I’m working on a project that should help math teachers, students, and mathematicians share high-quality math problems.
  • I’m working on a project that should lead to higher-quality education standards in all subject areas. I am hopeful that this project will lead to better tools for teachers, students, and parents.
  • I am getting ready to write a followup to Python Crash Course.
  • I’m watching how people use the Python Flash Cards I developed with No Starch Press. They haven’t sold particularly well online, but people love them when they see them in person. I’d like to publicize the ways that people are using these to solidify their understanding of Python.

Which Python libraries are your favorite (core or 3rd party)?

It might sound boring, but I love ReportLab. I would like to see a more open standard than PDF, but the reality is that PDFs are one of the easiest formats for people to share documents in. Unless you want to be an idealist rejecting any project that involves PDFs, sooner or later as a programmer you’re going to have to dig into PDF internals. I’ve worked on some really impactful projects that use PDFs in a critical role. When I’ve had to do that, ReportLab has made the work approachable. I did that before you wrote your book on ReportLab; now that I’m finished with classroom teaching I’m looking forward to working through your book.

I also love MatplotLib and Plotly. Matplotlib is the foundation for many modern Python plotting libraries, and I strive to maintain fluency with it and use it for a variety of projects. I always look for a friendly higher-level visualization library to work with as well, and right now Plotly fills that role well. I like that Plotly has ports in a variety of other languages as well. Requests has been really helpful in dealing with any network-based data source. I also love Jupyter notebooks for exploratory visualization work, and for educational projects as well.

I absolutely love Django. It’s a great time to become a programmer, and when I think about why that is, Django is one of the first things that comes to mind. When I was growing up, if you had an important program you were working on and you wanted to share it with the world, you needed to find a way to distribute physical copies of your code to the rest of the world. That was a daunting task, and really something only a professional programmer could do. Now if you have an idea and some basic programming skills, you can build a simple web app and have your project available to the entire world in a day. That’s incredible! Django isn’t really about building web sites; it’s about using a web site as a way to share your programming work with the rest of the world. I’m really looking forward to helping more people understand how to use Django to make their small projects available to the rest of the world quickly, and with confidence.

What inspired you to write “Python Crash Course”?

I spoke about this earlier, but I’ll add a little bit to that. It was really frustrating to try a wide variety of learning resources with high school students, and watch them not work for a variety of reasons. Things worked out for my students, because I had enough background knowledge to fill in the gaps for them. I recognized that independent learners, and teachers without a strong programming background, would probably abandon many of these resources and even give up on learning to program.

I decided to write a book that made no assumptions about the reader’s technical background, while respecting them as a mature learner.

What did you learn going through the writing / publishing process?

Oh my. It’s really true, you learn as much from being the teacher as you do from being the student. I have learned as much from writing Python Crash Course as people do from reading it.

Writing a book is a concrete project. You’re never really done writing a technical book, but there are really concrete milestones. It was really nice to have a project I could fully commit to, and know there would be a finished product I could hold at some point. That’s very different from classroom teaching. I learned about commitment, completion, professionalism, and deep humility. No book is ever perfect, especially a longer technical book. It is deeply humbling to have mistakes pointed out, and to have to stay vigilant for required updates.

Do you have any advice for aspiring writers?

First, it is every bit as much work as people say it is, and then some. I thought I’d need six months to write Python Crash Course, and it took almost two and a half years. Find a need of your own that you want to fulfill, and write to address that need.

Pay attention to the process. Learn to use a version control system like Git, and consider how to use it as a technical writer. Your commits should not be the same as they are when working purely as a programmer. For example every time I make a screenshot for a book, I make a commit with a message such as “Generated screenshot 15-4.” That way when the production editor asks for a series of updated screenshots, I can check out those commits and easily regenerate any screenshot at any time. Also, when developing new material, I write a program and make commits about any point of the development process I want to explain to the reader. This lets me focus on developing a meaningful, well-architected project, and then go back to focus on the writing later.

If you write a technical book and get it published, keep it updated! So many technical books go out of date more quickly than they need to. As an author, you can make minor corrections and updates every time your book goes through a new printing. The first edition of Python Crash Course went through about eleven printings. Almost every new printing involved some minor corrections or updates. For example, the Django project was originally written using Django 1.8. The latest printing of the first edition used Django 2.1. A huge part of the success of this book has been due to these updates. If I never updated the book, people would have criticized it as increasingly out of date a long time ago.

If you’re interested in writing a technical book, write online first. It’s a great way to practice writing, and to gain a following as someone who knows what they’re talking about on a specific topic. It’s also helpful for knowing how to develop and maintain a set of online resources for your book.

Finally, just read and write. Read non-technical books as well, so your technical writing isn’t too dry. Even a technical book tells a story, and good technical writers are good storytellers as well.

Is there anything else you’d like to say?

There’s a lot of talk today about everyone learning to code. I’m not convinced that everyone should learn how to code, but I am absolutely committed to the idea that everyone who wants to learn how to write code, should be able to. Tear down all the walls! Programming is power, and that power needs to be shared with almost everyone who wants it.

If you’d like to share your thoughts about Python Crash Course, I have a survey for gathering feedback. I’m using the results of the survey to continually refine the book, and to have a better sense of what other learning resources to develop.

If you’d like to know more about what it’s like to live on a bicycle for a year, I wrote about it in The Road to Alaska. It hasn’t received as much attention as my technical writing, but it’s a fun read if you’re into travel stories.

I’m @ehmatthes on Twitter, and if you want to get in touch feel free to email me at ehmatthes at gmail. Thanks everyone!

Thanks for doing the interview, Eric!

The post PyDev of the Week: Eric Matthes appeared first on The Mouse Vs. The Python.

PyBites: PyCon Australia 2019

$
0
0

PyCon Australia 2019 was, surprisingly, my first Australian Python Convention. It was also the first Python Convention I've attended purely as a spectator. I didn't contribute officially and was just there to learn and meet people.

What an experience!

The Talks

Let's get to the meat of it. I was impressed by the quality, topic range and professionalism of the talks. Almost all talks I attended, delivered a solid experience. There were very few I got bored in or was underwhelmed with.

Here are my top picks:


The unspeakable horror of discovering you didn't write tests - Melanie Crutchfield

Tests are tough. They're even tougher to get into. Melanie delivered a quality, educational and entertaining talk on why it's important to use them. Her story on how she came to use and appreciate tests made the talk relatable which kept me hooked. As a result, I'm actually super excited to get cracking with pytest, selenium and pytest-cov!

Takeaway:pytest-cov is an awesome tool to check what test coverage your code has. Very cool! Also, Melanie is pretty damn awesome and an incredibly entertaining presenter!

YouTube:https://www.youtube.com/watch?v=QD9YlNoTm2c

Slides/Notes: TBD


It's dark and my lights aren't working (an asyncio success story) - Jim Mussared

While the talk didn't end up diving too much into asyncio it more than made up for it by being damn interesting. Jim tells the story of his experience trying to automate the lights in his home. It's a story of hardship, dedication to the cause and is down right inspiring. Seriously, watch this one and see just how technical he gets with the technology he's playing with. I'll also add that he had us laughing for the majority of the talk!

Takeaway: Sometimes tech just doesn't work. The stubbornness to never give up is what makes us techies and programmers such an amazing group. Even when the odds are against you, keep searching, keep digging and you'll find a way.

YouTube:https://www.youtube.com/watch?v=grouP9QfdyA

Slides/Notes: Slides. Transcript is in the speaker notes of the first slide.


Shipping your first Python package and automating future publishing - Chris Wilcox

I really enjoyed this one as I've never had to package anything up to publish on PyPI before. I'm even still a little green when it comes to packaging in general so having Chris go through the whole setup.py and setup.cfg thing was great. He covers best practices, Tox and Nox as well.

Takeaway: While setup.py is still a thing, setup.cfg is much more flexible to use. I'll be seeing how I can utilise setup.cfg from now on. I also need to start diving into Tox (and Nox) to see what all the hubbub is about.

YouTube:https://www.youtube.com/watch?v=nietrteetKQ

Slides/Notes:Slides


Python Oddities Explained - Trey Hunner

This was easily one of my favourite talks. Trey has such a deep understanding of how Python works and has a wonderfully engaging presentation style. He covers a bunch of weird behaviours in Python that he's encountered over the years. It's a highly informative talk and I loved it.

Takeaway: There is a lot that can catch you out in Python. If you have any weird behaviour, have a good look through your code and make sure you have your scope referenced correctly. The += operator can be insanely useful and painful!

YouTube:https://www.youtube.com/watch?v=4MCT4WLf7Ac

Slides/Notes:Slides


Goodbye print statements, hello debugger! - Nina Zakharenko

Debugging my code has long been a fear of mine. At most, I use print() statements to help me see what's going on where and what object is being assigned what value. It's a pain, but it's comfortable and it works. Nina shattered the barrier into debugging for me. Using pdb and ipdb she demonstrated how to use the tool to debug code and managed to do so in a way that didn't make the complete noob (me) feel like an idiot. I've never been so excited for my code to break!

Takeaway:ipdb is going to be the debugger I jump into. I should also stop being so afraid to dive into a new topic when it comes to coding. Also, Microsoft Clippy still lives! (In sticker form).

YouTube:https://www.youtube.com/watch?v=HHrVBKZLolg

Slides/Notes:Slides and Blog


The People

I've said before that it's the people that make the conference special. PyCon AU was no different. I met so many incredible people over the 3 days. Some were even drawn to the @PythonicStaff I was carrying around in Anthony Shaw's absence.

Of special note, I have to mention two fantastic members of our PyBites Community - Anthony and Marc. Both of these chaps came out and allowed me to spend time with them throughout the conference.

Attending talks is always fun but doing it with friends (I'm assuming we're now friends) makes it a little more special. We learned a lot together and the guys were so inspired they even ran off to buy some MicroBit devices so they could play around with MicroPython.

Of course, PyBites Community aside, there were all sorts of cluey people around. Within 10 minutes of being at the conference I met a guy named Gavin who was doing audio analysis with an Adafruit device mounted to a home-made wooden box. (This is totally an understatement, what he was doing was way over my head!).

I was able to forge new relationships with Pythonistas from around Australia (including one who was taking the #100DaysofCode in Python course !) and I got to spend much needed time with friends I'd made at PyCon US. Trey and Melanie, you guys rock!


The Code

Some of the coding concepts, ideas and thoughts I've taken away from the conference:

  • I need to push having tests on all of my code. I'll be using pytest and pytest-cov for most code but will dive into Selenium where required. It's time to stop being intimidated and just start doing it.

  • On that same note, pdb or ipdb is going to start making an appearance. I want to debug properly.

  • Never leave pdbbreakpoint()s in Production Code! (Python 3.7)

  • As I've wanted to get into drones, I should take a look at pyparrot to add some Python goodness to the experience.

  • Give doctest a go as a testing framework. Could be useful in smaller situations where I don't want my code to be able to run unless it passes all tests.

  • It's way out of my area of expertise right now but WASM looks like it could be fun to dive into.

  • If I want to have a play with 3D Rendering in Python, I should take a look at vpython and glowscript.org.

  • Have a stab at using cookiecutter.

  • I really need to make a concerted effort to understand the "behind the scenes" of Python. Which functions call which dunder methods and how they work.

  • MicroPython is awesome.

  • Consider the impact to my users when I roll out updates. Something I already do but just a timely reminder from the "What PHP Learned from Python" talk.


The Missed Talks

There were quite a few talks I missed due to conflicts in the schedule, my ability to get to them or just plain being knackered:

All PyCon AU 2019 talks can be seen on YouTube here.


Inclusivity

It'd be remiss of me not to mention just how inclusive PyCon AU is. It doesn't matter what race, gender or however you identify, you'll have a home there.

Everyone was comfortable in their own skin. Everyone was respectful of opinions and appearance. Heck, the bathrooms even had powerful signage to ensure people were empowered to use whichever bathroom they felt comfortable using.

Seeing people of all shapes, sizes, colours, *, all learning harmoniously without any prejudice was truly a highlight for me. I couldn't be more proud to be part of this community.


Conclusion

As you can see, I missed quite a few quality talks but that's how it goes at PyCon. The chance to spend time with friends and to learn is one you should never pass up. There's so much to be gained from the PyCon conferences thanks largely to the amazing Python Community.

It's been 3 brilliant, insightful and inspirational days and I'm grateful for the time I was allowed there. I was even able to, finally, introduce one of my favourite colleagues to the community!

If you haven't been to a PyCon yet, make it happen. If you can get to one in Australia, even better. Just make sure you stop by and say G'day!

Keep Calm and Code in Python!

-- Julian

PSF GSoC students blogs: weeklyCheckIn[10]

$
0
0

What did I do this week?

As discussed in the earlier blog, I worked on tests for GUI to 'checkout' to a previous version of flightpath. After creating a pull request, I parallelly started working on GUI to add users and projects. I also got the pull request reviewed by my mentor.

What's coming up next?

This week, I have to finish up on modifications suggested by my mentor. After first request gets merged, I have to work on displaying user's, project's details in chat window, and active view windows. After that, I'd have to fix an issue due to which table view can't be created more than once for mscolab. Before making the second pull request, I also have to write tests for GUI part related to creation of users and projects.

Did I get stuck anywhere?

I was stuck for a very short while trying to decide if QInputDialog works with multiple inputs, then I designed my own QLabel for my usage.

PSF GSoC students blogs: Week 9 Chek-in

$
0
0

What did you do this week?

Documentation writing. The chapter for Crystallinity map calculation is written.

What is coming up next?

Chapter for Region clustering, Jupiter notebook for feature vector generation, non-zero order peaks detection and region clustering.

Did you get stuck anywhere?

Not yet.

Kushal Das: Adding directory to path in csh on FreeBSD

$
0
0

While I was trying to install rust on a FreeBSD box, I figured that I will have to update the path on the system with directory path of the ~/.cargo/bin. I added the following line in the ~/.cshrc file for the same.

set path = ( $path /home/kdas/.cargo/bin)

I am yet to learn much about csh, but, I can count this as a start.

PSF GSoC students blogs: Week-10: Caching in the pipeline

$
0
0

Hello folks,

This week was pretty tough. A seemingly simple task of setting up docs CD pipeline to execute notebooks ate up entire week, thanks to the complexities of DevOps! 🙄

What did I do this week?

  1. To execute our notebooks from docs CD pipeline, we need to make cached filter data available at the VM as a pre-build step. So I decided to store it at Azure as artifact (Universal Package) so that each time pipeline runs, it can instantly download artifact into VM and then also update it before using. Since it's updating the artifact, I decided to also publish it back to artifact feed thereat so that next time we get a more up-to-date data.
    • For 1st build of such a pipeline, I needed to make sure that artifact is already present in feed - which means publish the artifact without pipeline but from Azure CLI, locally. The authentication in Azure CLI was really cumbersome but I figured out how to use a PAT for it & then published the filter data as artifact.
    • Then I wrote script steps in pipeline to download artifact & bring it in right directory. The challenge here was to make pipeline download the latest versioned artifact from feed. After some mind-boggling research, I found how to achieve it with Azure REST API, but problem was again authentication while calling the API. On trying to solve it next day with a clear head, I found the solution that was right in front of my eyes in an already opened SO thread which I was overlooking!
    • I wrote script to conditionally publish the filter data as a newer versioned artifact if it got updated. Here I needed to write a python command that calls update function & returns update status back to bash script. But due to logger enabled in function, the returned value was entirely messed up - I fixed it up by disabling logger.
    • I also improved the versioning of artifacts by using date as version but there were conflicts with SemVer accepted by Azure, which again took time to manipulate date into acceptable version.
  2. Next I needed to make sure that executed notebooks give right outputs.
    • The matplotlib plots didn't appear in rendered noteboook. After some searching & digging, I found it was because we were interactively plotting graphs using %pylab notebook. By using %matplotlib inline magic which works non-interactively I made the plots appear.
    • I also cleaned some unnecessary data from quickstart notebook & made documentation more clear.

 

What is coming up next?

There's still a problem with pipeline, it is failing for PR builds although it works fine for my fork. I will try to fix that and then we will possibly move to starkit, where we can integrate it with wsynphot (on which I am currently working) to produce an interface for calculating photometery.

 

Did I get stuck anywhere?

Yes, it was these unexpected problems that took me finishing off this task of making pipeline execute the notebooks, an entire week! But I eventually solved all of them except that PR build failing problem.

 

What was something new I learned?

⚙️ This week made me learn many new things about Azure DevOps, like:

  • Azure CLI& how to use & authorize it to manage Azure resources from another (local) system
  • Azure REST API i.e. a really powerful API which lets you create/retrieve/update/delete the Azure resources
  • System.AccessToken which is a special predefined variable that used as OAuth token to access the REST API
  • Unlike powershell task which runs Windows PowerShell, we can use pwsh task on LINUX VMs since it runs PowerShell Core

🔡 While passing variable from a child process (python script) to parent shell (bash), make sure that you only write the value which you want to be passed on stdout. This means keep a check that there are no such function calls with print or logging statements in your script other than value you want to pass by printing it.

🧐 Openness for the strange options while researching: When we search for the solution of a problem on internet, lot of new & weird information comes before us which we just skim enough to decide that it is not for our case. But even then if we try to understand it, we may get how to make it work for our case by experimenting. Same happened when I was looking for how to authorize my build for Azure API call. On a SO thread I found a powershell script for it but I didn't bother to understand it thinking that powershell script can't be of any use to a LINUX VM. But when I eventually found the solution (SystemAcessToken & pwsh for LINUX), I was like: After all this time answer was right in front of my eyes and I was searching it here & there, by not caring to give it some minutes to understand! 

 


Thank you for reading. Stay tuned to learn about more interesting things with me!


PSF GSoC students blogs: Weekly blog #5 (week 10): 29/07 to 04/08

$
0
0

Hey. It looks like we’re in August now. Summer is coming to an end fast!

 

What happened this week? One of my mentors came back from holidays and we had a sync up call on Monday, and another one on Thursday (which should’ve been on Wednesday but I overslept. Set your alarms students!). We discussed the state of the project and what else can we do before GSoC ends.

 

At the start of the week I opened a PyTorch PR that supports both image and text explanations.

 

One thing that I was stuck on was saving and loading PyTorch models. Unlike in Keras where you can use one function call to load a model, in PyTorch there are multiple steps: get the class for the model, instantiate a model object, load the saved “state dict” into that new instance. In my case I was stuck on instantiating the model object, which required passing pre-trained word embeddings. My workaround was passing a tensor of zeros (instead of for example passing glove vectors, which were in the “state dict” anyways).

 

For the rest of the week I worked on the text PR. The first major progression was in adding integration tests. It took me some time to train and prepare two quality models from scratch, one for sentiment classification and another for multi-class classification.

 

Another development was in the documentation - function docstrings, library docs for text, adding new packages to the index, etc.

 

Manually testing the text PR, everything looked good apart from a few things. There was an issue with explaining “1D” layers (dense/fully-connected and final recurrent layers) that I fixed by adding special case code.

 

A remaining issue is with multi-label classification (https://keras.io/datasets/#reuters-newswire-topics-classification), where the explanations do not explain anything. One of my mentor’s suggested solutions was starting with a simple model and building from there.

 

These were the major issues and solutions for week 10. For week 11 I hope to follow this rough schedule:

 

Monday

  • In the text PR, move changes to the image API to a new PR. This way we can merge PR’s faster and possibly get early feedback.

Tuesday

  • Text PR remaining unit tests.
  • Text PR tutorial (now that I have two good models from the integration tests).

Wednesday

  • Text PR ad hoc tasks (FIXME’s, look at other implementations).
  • PyTorch PR refactor code (^).

Thursday

  • PyTorch PR tests (manual and automated?).
  • PyTorch PR docs, tutorial.

Friday

  • Contingency (^^^)

 

That’s all for the week. Thank you for reading!

Tomas Baltrunas

Sumana Harihareswara - Cogito, Ergo Sumana: Kickoff for Python 2 Sunsetting Communications Work

$
0
0
Python's 2.x line will reach End of Life on January 1, 2020, meaning that the maintainers of Python 2 will stop supporting it, even for security patches. Many institutions and codebases have not yet ported their code from Python 2 to Python 3. And many of them haven't even heard yet about the upcoming EOL. Volunteers have made many resources to help publicize and educate, but there's still more work to be done.

So the Python Software Foundation has contracted with my firm, Changeset Consulting, to help communicate about the sunsetting of Python 2. The high-level goal for Changeset's involvement is to help users through the end of the transition, help with communication so volunteers are not overwhelmed, and help update public-facing assets so core developers are not overwhelmed.

During this project (starting now and going till the end of April 2020), Changeset's goals are:

  • Create a concise page on python.org giving community guidance on the why and what of EOL
  • Create and publicize a formal press release through the PSF
  • Audit and update every mention of Python 2 on python.org, including in documentation and the wiki
  • Develop a plan to handle redirecting https://docs.python.org/2/* (especially deep links)
  • Publicize the new "what's up with EOL" page to a variety of audiences, respond to questions, and keep the page updated until PyCon US 2020 based on questions that come up

So, towards those goals, you'll see me with my colleagues:

  • Researching, writing, and editing technical documents and the press release
  • Corresponding with stakeholders, writing and publishing announcements and other communications in multiple media
  • Initiating and facilitating meetings (I will try to make them very minimal and short; if I invite you, please let me know your response)
  • Filing, triaging, labelling, prioritizing, and linking issues, such as in the website repo

For accountability, Changeset will provide reporting via:

I'm going to be asking for a lot of help along the way from the Python community: meeting with us, answering our questions, double-checking our drafts for accuracy, publicizing that EOL page to your circles, setting up some parties for January 1st. Thanks in advance and let's get the Python user base further along towards enjoying the shininess of Python 3.

Catalin George Festila: Python 3.7.3 : Using the flask - part 009.

$
0
0
In this tutorial, I will show you how to use blueprints with a new type of flask project using multiple python files. I show you just how to create the A.P.I. not the REACT front end. Flask uses a concept of blueprints for making application components and supporting common patterns within an application or across applications. Blueprints can greatly simplify how large applications work and

Catalin George Festila: Python 3.7.3 : Using the flask - part 006.

$
0
0
Today I will show you how to use the RESTful API application with flask python module that uses HTTP requests to GET, PUT, POST and DELETE data. When HTTP is used, as is most common, the operations (HTTP methods) available are GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE.[2], see the Wikipedia article. All of these HTTP methods will be tested with postman software. You need to

Talk Python to Me: #224 12 lessons from 100 days of web

$
0
0
Back in May of 2018, Bob Belderbos, Julian Sequeira, and I started on what would be a 9-month project. We wanted to create a dedicated, 100 days of code course specifically for Python web developers. Much of what we created for that course, we had prior experience with. But much of it was also new to us.

PSF GSoC students blogs: Week-10: Executing notebooks from the Pipeline

$
0
0

Hello folks,

This week was pretty tough. A seemingly simple task of setting up docs CD pipeline to execute notebooks ate up entire week, thanks to the complexities of DevOps! 🙄

What did I do this week?

  1. To execute our notebooks from docs CD pipeline, we need to make cached filter data available at the VM as a pre-build step. So I decided to store it at Azure as artifact (Universal Package) so that each time pipeline runs, it can instantly download artifact into VM and then also update it before using. Since it's updating the artifact, I decided to also publish it back to artifact feed thereat so that next time we get a more up-to-date data.
    • For 1st build of such a pipeline, I needed to make sure that artifact is already present in feed - which means publish the artifact without pipeline but from Azure CLI, locally. The authentication in Azure CLI was really cumbersome but I figured out how to use a PAT for it & then published the filter data as artifact.
    • Then I wrote script steps in pipeline to download artifact & bring it in right directory. The challenge here was to make pipeline download the latest versioned artifact from feed. After some mind-boggling research, I found how to achieve it with Azure REST API, but problem was again authentication while calling the API. On trying to solve it next day with a clear head, I found the solution that was right in front of my eyes in an already opened SO thread which I was overlooking!
    • I wrote script to conditionally publish the filter data as a newer versioned artifact if it got updated. Here I needed to write a python command that calls update function & returns update status back to bash script. But due to logger enabled in function, the returned value was entirely messed up - I fixed it up by disabling logger.
    • I also improved the versioning of artifacts by using date as version but there were conflicts with SemVer accepted by Azure, which again took time to manipulate date into acceptable version.
  2. Next I needed to make sure that executed notebooks give right outputs.
    • The matplotlib plots didn't appear in rendered noteboook. After some searching & digging, I found it was because we were interactively plotting graphs using %pylab notebook. By using %matplotlib inline magic which works non-interactively I made the plots appear.
    • I also cleaned some unnecessary data from quickstart notebook & made documentation more clear.

 

What is coming up next?

There's still a problem with pipeline, it is failing for PR builds although it works fine for my fork. I will try to fix that and then we will possibly move to starkit, where we can integrate it with wsynphot (on which I am currently working) to produce an interface for calculating photometery.

 

Did I get stuck anywhere?

Yes, it was these unexpected problems that took me finishing off this task of making pipeline execute the notebooks, an entire week! But I eventually solved all of them except that PR build failing problem.

 

What was something new I learned?

⚙️ This week made me learn many new things about Azure DevOps, like:

  • Azure CLI& how to use & authorize it to manage Azure resources from another (local) system
  • Azure REST API i.e. a really powerful API which lets you create/retrieve/update/delete the Azure resources
  • System.AccessToken which is a special predefined variable that used as OAuth token to access the REST API
  • Unlike powershell task which runs Windows PowerShell, we can use pwsh task on LINUX VMs since it runs PowerShell Core
  • Conditionally run a step in Azure by using conditions option.

🔡 While passing variable from a child process (python script) to parent shell (bash), make sure that you only write the value which you want to be passed on stdout. This means keep a check that there are no such function calls with print or logging statements in your script other than value you want to pass by printing it.

🧐 Openness for the strange options while researching: When we search for the solution of a problem on internet, lot of new & weird information comes before us which we just skim enough to decide that it is not for our case. But even then if we try to understand it, we may get how to make it work for our case by experimenting. Same happened when I was looking for how to authorize my build for Azure API call. On a SO thread I found a powershell script for it but I didn't bother to understand it thinking that powershell script can't be of any use to a LINUX VM. But when I eventually found the solution (SystemAcessToken & pwsh for LINUX), I was like: After all this time answer was right in front of my eyes and I was searching it here & there, by not caring to give it some minutes to understand! 

 


Thank you for reading. Stay tuned to learn about more interesting things with me!

Philippe Normand: Review of the Igalia Multimedia team Activities (2019/H1)

$
0
0

This blog post takes a look back at the various Multimedia-related tasks the Igalia Multimedia team was involved in during the first half of 2019.

GStreamer Editing Services

Thibault added support for the OpenTimelineIO open format for editorial timeline information. Having many editorial timeline information formats supported by OpenTimelineIO reduces …


Real Python: What You Need to Know to Manage Users in Django Admin

$
0
0

User management in Django admin is a tricky subject. If you enforce too many permissions, then you might interfere with day-to-day operations. If you allow for permissions to be granted freely without supervision, then you put your system at risk.

Django provides a good authentication framework with tight integration to Django admin. Out of the box, Django admin does not enforce special restrictions on the user admin. This can lead to dangerous scenarios that might compromise your system.

Did you know staff users that manage other users in the admin can edit their own permissions? Did you know they can also make themselves superusers? There is nothing in Django admin that prevents that, so it’s up to you!

By the end of this tutorial, you’ll know how to protect your system:

  • Protect against permission escalation by preventing users from editing their own permissions
  • Keep permissions tidy and maintainable by only forcing users to manage permissions only using groups
  • Prevent permissions from leaking through custom actions by explicitly enforcing the necessary permissions

Follow Along:

To follow along with this tutorial, it’s best to setup a small project to play with. If you aren’t sure how to do that, then check out Get Started With Django.

This tutorial also assumes a basic understanding of user management in Django. If you aren’t familiar with that, then check out the official documentation.

Free Bonus:Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

Model Permissions

Permissions are tricky. If you don’t set permissions, then you put your system at risk of intruders, data leaks, and human errors. If you abuse permissions or use them too much, then you risk interfering with day-to-day operations.

Django comes with a built-in authentication system. The authentication system includes users, groups, and permissions.

When a model is created, Django will automatically create four default permissions for the following actions:

  1. add: Users with this permission can add an instance of the model.
  2. delete: Users with this permission can delete an instance of the model.
  3. change: Users with this permission can update an instance of the model.
  4. view: Users with this permission can view instances of this model. This permission was a much anticipated one, and it was finally added in Django 2.1.

Permission names follow a very specific naming convention: <app>.<action>_<modelname>.

Let’s break that down:

  • <app> is the name of the app. For example, the User model is imported from the auth app (django.contrib.auth).
  • <action> is one of the actions above (add, delete, change, or view).
  • <modelname> is the name of the model, in all lowercase letters.

Knowing this naming convention can help you manage permissions more easily. For example, the name of the permission to change a user is auth.change_user.

How to Check Permissions

Model permissions are granted to users or groups. To check if a user has a certain permission, you can do the following:

>>>
>>> fromdjango.contrib.auth.modelsimportUser>>> u=User.objects.create_user(username='haki')>>> u.has_perm('auth.change_user')False

It’s worth mentioning that .has_perm() will always return True for active superuser, even if the permission doesn’t really exist:

>>>
>>> fromdjango.contrib.auth.modelsimportUser>>> superuser=User.objects.create_superuser(... username='superhaki',... email='me@hakibenita.com',... password='secret',)>>> superuser.has_perm('does.not.exist')True

As you can see, when you’re checking permissions for a superuser, the permissions are not really being checked.

How to Enforce Permissions

Django models don’t enforce permissions themselves. The only place permissions are enforced out of the box by default is Django Admin.

The reason models don’t enforce permissions is that, normally, the model is unaware of the user performing the action. In Django apps, the user is usually obtained from the request. This is why, most of the time, permissions are enforced at the view layer.

For example, to prevent a user without view permissions on the User model from accessing a view that shows user information, do the following:

fromdjango.core.exceptionsimportPermissionDenieddefusers_list_view(request):ifnotrequest.user.has_perm('auth.view_user'):raisePermissionDenied()

If the user making the request logged in and was authenticated, then request.user will hold an instance of User. If the user did not login, then request.user will be an instance of AnonymousUser. This is a special object used by Django to indicate an unauthenticated user. Using has_perm on AnonymousUser will always return False.

If the user making the request doesn’t have the view_user permission, then you raise a PermissionDenied exception, and a response with status 403 is returned to the client.

To make it easier to enforce permissions in views, Django provides a shortcut decorator called permission_required that does the same thing:

fromdjango.contrib.auth.decoratorsimportpermission_required@permission_required('auth.view_user')defusers_list_view(request):pass

To enforce permissions in templates, you can access the current user permissions through a special template variable called perms. For example, if you want to show a delete button only to users with delete permission, then do the following:

{%ifperms.auth.delete_user%}<button>Delete user!</button>{%endif%}

Some popular third party apps such as the Django rest framework also provide useful integration with Django model permissions.

Django Admin and Model Permissions

Django admin has a very tight integration with the built-in authentication system, and model permissions in particular. Out of the box, Django admin is enforcing model permissions:

  • If the user has no permissions on a model, then they won’t be able to see it or access it in the admin.
  • If the user has view and change permissions on a model, then they will be able to view and update instances, but they won’t be able to add new instances or delete existing ones.

With proper permissions in place, admin users are less likely to make mistakes, and intruders will have a harder time causing harm.

Implement Custom Business Roles in Django Admin

One of the most vulnerable places in every app is the authentication system. In Django apps, this is the User model. So, to better protect your app, you are going to start with the User model.

First, you need to take control over the User model admin page. Django already comes with a very nice admin page to manage users. To take advantage of that great work, you are going to extend the built-in User admin model.

Setup: A Custom User Admin

To provide a custom admin for the User model, you need to unregister the existing model admin provided by Django, and register one of your own:

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin# Unregister the provided model adminadmin.site.unregister(User)# Register out own model admin, based on the default UserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):pass

Your CustomUserAdmin is extending Django’s UserAdmin. You did that so you can take advantage of all the work already done by the Django developers.

At this point, if you log into your Django admin at http://127.0.0.1:8000/admin/auth/user, you should see the user admin unchanged:

Django bare boned user admin

By extending UserAdmin, you are able to use all the built-in features provided by Django admin.

Prevent Update of Fields

Unattended admin forms are a prime candidate for horrible mistakes. A staff user can easily update a model instance through the admin in a way the app does not expect. Most of the time, the user won’t even notice something is wrong. Such mistakes are usually very hard to track down and fix.

To prevent such mistakes from happening, you can prevent admin users from modifying certain fields in the model.

If you want to prevent any user, including superusers, from updating a field, you can mark the field as read only. For example, the field date_joined is set when a user registers. This information should never be changed by any user, so you mark it as read only:

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):readonly_fields=['date_joined',]

When a field is added to readonly_fields, it will not be editable in the admin default change form. When a field is marked as read only, Django will render the input element as disabled.

But, what if you want to prevent only some users from updating a field?

Conditionally Prevent Update of Fields

Sometimes it’s useful to update fields directly in the admin. But you don’t want to let any user do it: you want to allow only superusers to do it.

Let’s say you want to prevent non-superusers from changing a user’s username. To do that, you need to modify the change form generated by Django, and disable the username field based on the current user:

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):defget_form(self,request,obj=None,**kwargs):form=super().get_form(request,obj,**kwargs)is_superuser=request.user.is_superuserifnotis_superuser:form.base_fields['username'].disabled=Truereturnform

Let’s break it down:

  • To make adjustments to the form, you override get_form(). This function is used by Django to generate a default change form for a model.
  • To conditionally disable the field, you first fetch the default form generated by Django, and then if the user is not a superuser, disable the username field.

Now, when a non-superuser tries to edit a user, the username field will be disabled. Any attempt to modify the username through Django Admin will fail. When a superuser tries to edit the user, the username field will be editable and behave as expected.

Prevent Non-Superusers From Granting Superuser Rights

Superuser is a very strong permission that should not be granted lightly. However, any user with a change permission on the User model can make any user a superuser, including themselves. This goes against the whole purpose of the permission system, so you want to close this hole.

Based on the previous example, to prevent non-superusers from making themselves superusers, you add the following restriction:

fromtypingimportSetfromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):defget_form(self,request,obj=None,**kwargs):form=super().get_form(request,obj,**kwargs)is_superuser=request.user.is_superuserdisabled_fields=set()# type: Set[str]ifnotis_superuser:disabled_fields|={'username','is_superuser',}forfindisabled_fields:iffinform.base_fields:form.base_fields[f].disabled=Truereturnform

In addition to the previous example, you made the following additions:

  1. You initialized an empty set disabled_fields that will hold the fields to disable. set is a data structure that holds unique values. It makes sense to use a set in this case, because you only need to disable a field once. The operator |= is used to perform an in-place OR update. For more information about sets, check out Sets in Python.

  2. Next, if the user is a superuser, you add two fields to the set (username from the previous example, and is_superuser). They will prevent non-superusers from making themselves superusers.

  3. Lastly, you iterate over the fields in the set, mark all of them as disabled, and return the form.

Django User Admin Two-Step Form

When you create a new user in Django admin, you go through a two-step form. In the first form, you fill in the username and password. In the second form, you update the rest of the fields.

This two-step process is unique to the User model. To accommodate this unique process, you must verify that the field exists before you try to disable it. Otherwise, you might get a KeyError. This is not necessary if you customize other model admins.

For more information about KeyError, check out Python KeyError Exceptions and How to Handle Them.

Grant Permissions Only Using Groups

The way permissions are managed is very specific to each team, product, and company. I found that it’s easier to manage permissions in groups. In my own projects, I create groups for support, content editors, analysts, and so on. I found that managing permissions at the user level can be a real hassle. When new models are added, or when business requirements change, it’s tedious to update each individual user.

To manage permissions only using groups, you need to prevent users from granting permissions to specific users. Instead, you want to only allow associating users to groups. To do that, disable the field user_permissions for all non-superusers:

fromtypingimportSetfromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):defget_form(self,request,obj=None,**kwargs):form=super().get_form(request,obj,**kwargs)is_superuser=request.user.is_superuserdisabled_fields=set()# type: Set[str]ifnotis_superuser:disabled_fields|={'username','is_superuser','user_permissions',}forfindisabled_fields:iffinform.base_fields:form.base_fields[f].disabled=Truereturnform

You used the exact same technique as in the previous sections to implement another business rule. In the next sections, you’re going to implement more complex business rules to protect your system.

Prevent Non-Superusers From Editing Their Own Permissions

Strong users are often a weak spot. They possess strong permissions, and the potential damage they can cause is significant. To prevent permission escalation in case of intrusion, you can prevent users from editing their own permissions:

fromtypingimportSetfromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):defget_form(self,request,obj=None,**kwargs):form=super().get_form(request,obj,**kwargs)is_superuser=request.user.is_superuserdisabled_fields=set()# type: Set[str]ifnotis_superuser:disabled_fields|={'username','is_superuser','user_permissions',}# Prevent non-superusers from editing their own permissionsif(notis_superuserandobjisnotNoneandobj==request.user):disabled_fields|={'is_staff','is_superuser','groups','user_permissions',}forfindisabled_fields:iffinform.base_fields:form.base_fields[f].disabled=Truereturnform

The argument obj is the instance of the object you are currently operating on:

  • When obj is None, the form is used to create a new user.
  • When obj is not None, the form is used to edit an existing user.

To check if the user making the request is operating on themselves, you compare request.user with obj. Because this is the user admin, obj is either an instance of User, or None. When the user making the request, request.user, is equal to obj, then it means that the user is updating themselves. In this case, you disable all sensitive fields that can be used to gain permissions.

The ability to customize the form based on the object is very useful. It can be used to implement elaborate business roles.

Override Permissions

It can sometimes be useful to completely override the permissions in Django admin. A common scenario is when you use permissions in other places, and you don’t want staff users to make changes in the admin.

Django uses hooks for the four built-in permissions. Internally, the hooks use the current user’s permissions to make a decision. You can override these hooks, and provide a different decision.

To prevent staff users from deleting a model instance, regardless of their permissions, you can do the following:

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):defhas_delete_permission(self,request,obj=None):returnFalse

Just like with get_form(), obj is the instance you currently operate on:

  • When obj is None, the user requested the list view.
  • When obj is not None, the user requested the change view of a specific instance.

Having the instance of the object in this hook is very useful for implementing object-level permissions for different types of actions. Here are other use cases:

  • Preventing changes during business hours
  • Implementing object-level permissions

Restrict Access to Custom Actions

Custom admin actions require special attention. Django is not familiar with them, so it can’t restrict access to them by default. A custom action will be accessible to any admin user with any permission on the model.

To illustrate, add a handy admin action to mark multiple users as active:

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):actions=['activate_users',]defactivate_users(self,request,queryset):cnt=queryset.filter(is_active=False).update(is_active=True)self.message_user(request,'Activated {} users.'.format(cnt))activate_users.short_description='Activate Users'# type: ignore

Using this action, a staff user can mark one or more users, and activate them all at once. This is useful in all sorts of cases, such as if you had a bug in the registration process and needed to activate users in bulk.

This action updates user information, so you want only users with change permissions to be able to use it.

Django admin uses an internal function to get actions. To hide activate_users() from users without change permission, override get_actions():

fromdjango.contribimportadminfromdjango.contrib.auth.modelsimportUserfromdjango.contrib.auth.adminimportUserAdmin@admin.register(User)classCustomUserAdmin(UserAdmin):actions=['activate_users',]defactivate_users(self,request,queryset):assertrequest.user.has_perm('auth.change_user')cnt=queryset.filter(is_active=False).update(is_active=True)self.message_user(request,'Activated {} users.'.format(cnt))activate_users.short_description='Activate Users'# type: ignoredefget_actions(self,request):actions=super().get_actions(request)ifnotrequest.user.has_perm('auth.change_user'):delactions['activate_users']returnactions

get_actions() returns an OrderedDict. The key is the name of the action, and the value is the action function. To adjust the return value, you override the function, fetch the original value, and depending on the user permissions, remove the custom action activate_users from the dict. To be on the safe side, you assert the user permission in the action as well.

For staff users without change_user() permissions, the action activate_users will not appear in the actions dropdown.

Conclusion

Django admin is a great tool for managing a Django project. Many teams rely on it to stay productive in managing day-to-day operations. If you use Django admin to perform operations on models, then it’s important to be aware of permissions. The techniques described in this article are useful for any model admin, not just the User model.

In this tutorial, you protected your system by making the following adjustments in Django Admin:

  • You protected against permission escalation by preventing users from editing their own permissions.
  • You kept permissions tidy and maintainable by only forcing users to manage permissions only using groups.
  • You prevented permissions from leaking through custom actions by explicitly enforcing the necessary permissions.

Your User model admin is now much safer than when you started!


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

qutebrowser development blog: Happy birthday, qutebrowser!

$
0
0

5 years ago today, this happened:

commit 11a94957dc038fc27c5ff976197ad2b2d0352d20
Author: Florian Bruhin <git@the-compiler.org>
Date:   Sat Dec 14 22:15:16 2013 +0100

    Initial commit

That's how qutebrowser looked a day after that (and that commit still seems to run!): https://imgur.com/a/xoG1r4G

Exactly a year later, things …

PSF GSoC students blogs: Time is running out - 5th blog post

$
0
0

Hello everyone.

 

The time is finally getting into me. I feel I am really late on everything I planned, and I know I am the one to blame.

This week we finally merged my code for the command setup, and with it we had tests, documentation and the fix of the first issues.

What mostly got into me was the lack of knowledge about how to fulfill a full project. I underestimated the tasks, and didn't pay attention to all the details it was necessary for my project to be 100%. This serves as learning, as now I am way more confident about planning, coding and delivering a full-realized project.

Delays are expected; that's why I left a 3-weeks time for writing and polishing before the end of the period. What I didn't expect was that I had so much to learn and my motivation dropping together with my university exams week. Today I feel more motivated, as one step of the project was fulfilled, and I know I still have a chance to do better until the end of the program. That's where my focus will be.

Writing code is not only about knowing the tools, but creativity and time commitment. Thinking is crucial for a good code, but also is having a good pace on how you're writing. Keeping communication with my mentors was hard to me, but within the period of 2 weeks of good communication I finally got back on track.

I expect to be way more positive in my next blog post :-)

And with more PRs merged!

 

Leonardo Rodrigues.

Codementor: How to automate artistic taste (Basic clustering example with python and sklearn)

$
0
0
Cellular automata patterns are clustered based on features related to beauty such as fractal dimensionality and compression efficiency in order to find out the most interesting ones.

Erik Marsja: How to Make a Scatter Plot in Python using Seaborn

$
0
0

Data visualization is a big part of the process of data analysis. In this post, we will learn how make a scatter plot using Python and the package Seaborn. In detail, we will learn how to use the Seaborn methods scatterplot, regplot, lmplot, and pairplot to create scatter plots in Python.

More specifically, we will learn how to make scatter plots, change the size of the dots, change the markers, the colors, and change the number of ticks.  Furthermore, we will learn how to plot a regression line, add text, plot a distribution on a scatter plot, among other things. Finally, we will also learn how to save Seaborn plots in high resolution. That is, we learn how to make print-ready plots.

Scatter plots are powerful data visualization tools that can reveal a lot of information. Thus, this Python scatter plot tutorial will start explain what they are and when to use them. After we done that, we will learn how to make scatter plots.

What is a Scatter Plot

A scatter plot is a two-dimensional (bivariate) data visualization that uses dots to represent the values gathered for two different variables. That is, one of the variables is plotted along the x-axis and the other plotted along the y-axis. For example, this scatter plot shows the relationship between a child’s height and the parents height.

When to Use a Scatter Plot

Scatter plots are used when we want to visualize the relationship between two variables. Furthermore, scatter plots are sometimes called correlation plots because they reveal how two variables are correlated. In the child and parent height example, the chart wasn’t just a simple log of the height of a set of children and parents, but it also visualized the relationship between a child’s height and it’s parents height. That is, namely that the child’s height increases as the parents height increases. Notice that the relationship isn’t perfect, some children are shorter than their parents. However, this trend is general and relatively pretty strong. Using Python and Seaborn we can also visualize this trend and correlation:

Python Scatter Plot Tutorial

This is exactly what we are going to learn in this tutorial; how to make a scatter plot using Python and Seaborn.

In the first code chunk, below, we are importing the packages we are going to use. Furthermore, to get data to visualize (i.e., create scatter plots) we load this from an URL. Note, the %matplotlib inline code is only needed if we want to display the scatter plots in a Jupyter Notebook. This is also the case for the import warnings and warnings.filterwarnings(‘ignore’) part of the code.

%matplotlib inline
import numpy as np
import pandas as pd
import seaborn as sns

# Suppress warningsimport warnings
warnings.filterwarnings('ignore')

# Import Data
data = 'https://vincentarelbundock.github.io/Rdatasets/csv/datasets/mtcars.csv'
df = pd.read_csv(data, index_col=0)

df.head()

How to use Seaborn to create Scatter Plots in Python

In this section we will learn how to make scatter plots using Seaborns scatterplot, regplot, lmplot, and pairplot methods.

Seaborn Scatter plot using the scatterplot method

First, we start with the most obvious method to create scatter plots using Seaborn: using the scatterplot method. In the first Seaborn scatter plot example, below, we plot the variables wt (x-axis) and mpg (y-axis). This will give us a simple scatter plot:

sns.scatterplot(x='wt', y='mpg', data=df)

Size of the dots

If we want to have the size of the dots represent one of the variables this is possible.  In the next example, we change the size of the dots using the size argument.

sns.scatterplot(x='wt', y='mpg',
                size='wt', data=df)

Change the Axis

In the next Seaborn scatter plot example we are going to learn how to change the ticks on the x- and y-axis’. That is, we are going to change the number of ticks on each axis. To do this we use the set methods and use the arguments xticks and yticks:

ax = sns.scatterplot(x='wt', y='mpg', size='wt', data=df)
ax.set(xticks=np.arange(1, 6, 1),
      yticks=np.arange(5, 50, 10))

Grouped Scatter Plot in Seaborn

If we have a categorical variable (i.e., a factor) and want to group the dots in the scatter plot we use the hue argument:

sns.scatterplot(x='wt', y='mpg',
                hue='vs', data=df)

Changing the Markers (the dots)

Now, one way to change the look of the markers is to use the style argument.

data = 'https://vincentarelbundock.github.io/Rdatasets/csv/carData/Burt.csv'
df = pd.read_csv(data, index_col=0)
sns.scatterplot(x='IQbio', y='IQfoster', style='class',
                hue='class', data=df)

Note, should also be possible to use the markers argument. However, it seems not to work, right now.

Bivariate Distribution on Scatter plot

In the next Seaborn scatter plot example we are going to plot a bivariate distribution on top of our scatter plot. This is quite simple and after we have called the scatterplot method we use the kdeplot method:

data = 'https://vincentarelbundock.github.io/Rdatasets/csv/datasets/mtcars.csv'
df = pd.read_csv(data, index_col=0)

sns.scatterplot(x='wt', y='mpg', data=df)
sns.kdeplot(df.wt, df.mpg)

Seaborn Scatter plot using the regplot method

If we want a regression line (trend line) plotted on our scatter plot we can also use the Seaborn method regplot. In the first example, using regplot, we are creating a scatter plot with a regression line. Here, we also get the 95% confidence interval:

sns.regplot(x='wt', y='mpg', data=df)

Scatter Plot Without Confidence Interval

We can also choose to create a scatter plot without the confidence interval. This is simple, we just add the ci argument and set it to None:

sns.regplot(x='wt', y='mpg',
            ci=None, data=df)

Note, the ci argument can also take a numerical value. That is, if we want a plot with 70% confidence interval we just change the ci=None to ci=70.

Seaborn regplot Without Regression Line

Furthermore, it’s possible to create a scatter plot without the regression line using the regplot method. In the next example, we just add the argument reg_fit and set it to False:

sns.regplot(x='wt', y='mpg',
            fit_reg=False, data=df)

Changing the Number of Ticks

It’s also possible to change the number of ticks when working with Seaborn regplot. In fact, it’s as simple as working with the scatterplot method, we used earlier. To change the ticks we use the set method and the xticks and yticks arguments:

ax = sns.regplot(x='wt', y='mpg', data=df)
ax.set(xticks=np.arange(1, 6, 1),
      yticks=np.arange(5, 50, 10))

Changing the Color on the Scatterplot

If we want to change the color of the plot, it’s also possible. We just have to use the color argument. In the regplot example below we are getting a green trend line and green dots:

sns.regplot(x='wt', y='mpg',
            color='g', data=df) 

Adding Text to a Seaborn Plot

In the next example, we are going to learn how to add text to a Seaborn plot. Remember the plot, in the beginning, showing the relationship between children’s height and their parents height? In that plot there was a regression line as well as the correlation coeffienct and p-value. Here we are going to learn how to add this to a scatter plot. First, we start by using pearsonr to calculate the correlation between the variables wt and mpg:

from scipy.stats import pearsonr

corr = pearsonr(df['wt'], df['mpg'])
corr = [np.round(c, 2) for c in corr]
print(corr)
# Output: [-0.87, 0.0]

In the next code chunk we are creating a string variable getting our pearson r and p-value (corr[0] and corr[1], respectively). Next, we create the scatter plot with the trend line and, finally, add the text. Note, the first tow arguments are the coordinates where we want to put the text:

text = 'r=%s, p=%s' % (corr[0], corr[1])
ax = sns.regplot(x="wt", y="mpg", data=df)
ax.text(4, 30, text, fontsize=12)

Seaborn Scatter Plot with Trend Line and Density

It’s, of course, also possible to plot a bivariate distrubiton on the scatter plot with a regression line. In the final example, using Seaborn regplot we just add the kernel density estimation using Seaborn kde method:

import seaborn as sns

sns.regplot(x='wt', y='mpg',
            ci=None, data=df)
sns.kdeplot(df.wt, df.mpg)

How to Rotate the Axis Labels on a Seaborn Plot

If the labels on the axis’ are long, we may want to rotate them for readability. It’s quite simple to rotate the axis labels on the regplot object. Here we rotatelabels on the x-axis 90 degrees:

ax = sns.regplot(x='wt', y='mpg', data=df)
for item in ax.get_xticklabels():
    item.set_rotation(90)

Scatter Plot Using Seaborn lmplot Method

In this section we are going to cover Seaborn’s lmplot method. This method enables us create grouped Scatter plots with a regression line.

In the first example, we are just creating a Scatter plot with the trend lines for each group. Note, here we use the argument hue to set what varible we want to group by.

sns.lmplot(x='wt', y='mpg',
           hue='vs', data=df)

Changing the Color Using the Palette Argument

When working with Seaborn’s lmplot method we can also change the color of the lines, CIs, and dots using the palette argument. See here for different palettes to use.

sns.lmplot(x='wt', y='mpg', hue='vs',
           palette="Set1", data=df)

Change the Markers on the lmplot

As previously mentioned, when creating scatter plots, the markers can be changed using the markers argument. In the next Seaborn lmplot example we are changing the markers to crosses:

sns.lmplot(x='wt', y='mpg',
           hue='vs', palette="Set1", markers='+', data=df)

Furthermore, it’s also possible to change the markers of two, or more, categories. In the example below we use a list with the cross and “star” and get a plot with crosses and stars, for each gorup.

sns.lmplot(x='wt', y='mpg',
           hue='vs',
           palette="Set1", markers=['+', '*'], data=df)

See here for different markers that can be used with Seaborn plots.

Seaborn Pairplot: Scatterplot + Histogram

In the last section, before learning how to save high resolution images, we are going to use Seaborn’s pairplot method. The pairplot method will create pairs of plots

data = 'https://vincentarelbundock.github.io/Rdatasets/csv/datasets/mtcars.csv'
df = pd.read_csv(data, index_col=0)
df = dfdf[['mpg', 'wt', 'hp', 'qsec']
sns.pairplot(df)
Pairplot with Grouped Data

If we want to group the data, with different colors for different categories, we can use the argument hue, also with pairplot. In this Python data visualization example we also use the argument vars with a list to select which variables we want to visualize:

cols = ['mpg', 'wt', 'hp', 'qsec']
sns.pairplot(df, vars=cols, hue='vs')

Saving a High Resolution Plot in Python

If we are planning to publish anything we need to save the data visualization(s) to a high resolution file. In the last code example, below, we will learn how to save a high resolution image using Python and matplotlib.

ax = sns.pairplot(df, vars=cols, hue='vs')
plt.savefig('pairplot.eps', format='eps', dpi=300)

Here’s a link to a Jupyter Notebook containing all of the above code.

Conclusion

In this post we have larned how to make scatter plots in Seaborn. Moreover, we have also learned how to:

  • work with the Seaborn’s scatterplot, regplot, lmplot, and pairplot methods
  • change color, number of axis ticks, the markers, and how to rotate the axis labels of Seaborn plots
  • save a high resolution, and print ready, image of a Seaborn plot

The post How to Make a Scatter Plot in Python using Seaborn appeared first on Erik Marsja.

Viewing all 22875 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>