Repository for running jupyter notebooks and keeping relevant files in one place
Updates from
documentation for different repositories
[closed], W., Wencel, W. and Agrawal, S. (2016) What is the difference between a feature and a label?, Stack Overflow. Available at: https://stackoverflow.com/questions/40898019/what-is-the-difference-between-a-feature-and-a-label#:~:text=Briefly%2C%20feature%20is%20input%3B%20label,region%2C%20family%20income%2C%20etc. (Accessed: 9 February 2023).
QUICKSTART-WIN-VSC-BASH.md from
Repository for running jupyter notebooks and keeping relevant files in one place
Repository for running jupyter notebooks and keeping relevant files in one place
check how to remove null values from dataframe
pandas .iloc() - locate by row, col indices .loc() - locate by row index and col NAME
df.drop('Column name', axis=1) - where axies = 0 for rows, 1 for columns - drops referenced column from data frame - inplace=True argument to ensure column stays dropped. df.drop(1,axis=0).reset_index() - new col with old indices df.drop(1,axis=0).reset_index(drop=True,inplace=True)
df.copy
df['columnname'].apply(type).value_counts() - this looks at and notes the values by type and then counts them
df['colname'] = df['colname'].replace('missing','inf'],np.nan) - replaces our specified strings 'missing' and 'inf' - with np.nan
df['colname'] = df['colname'].astype(float) - convert values to float
Review note: when we substitute np.nan in for strings the resulting data type is (if all the other entries are say float) float.
df.info() - rerunning this after data cleaning may result in cleaned columns type changing to, say, float.
Check length of each column Columns shorter than max col length means missing values as empty cells
data_new = pd.read_csv('/content/drive/MyDrive/Python Course/Melbourne_Housing.csv',na_values=['missing','inf'])
- on load, above line automatically converts all missing and inf to nan so, running: data_new['BuildingArea'].dtype
- gives dtype('float64') as only float (and nan which seems to be treated as whatever the rest of the data types are)
data['BuildingArea'].unique()
- above line run before cleaning gives unique values in column as a numpy array
- so can inspect to find out which strings to remove.
python3 -m venv .venv - in bash - and on Windows source .venv/bin/activate - in bash source .venv/Scripts/activate - on Windows - on VSCode Windows bash /workspace/machine-learning-classification/.venv/bin/python -m pip install --upgrade pip - in GitPod python3 -m pip install --upgrade pip - on Windows
.venv/Scripts/python.exe -m pip install --upgrade pip - in .venv
pip install --upgrade pip pip install jupyter notebook pip install matplotlib pip install pandas pip install seaborn pip install numpy pip install scipy pip install statsmodels pip install -U scikit-learn pip install ipykernel pip install nb-black
Ctrl Shift P Create New Jupyter Notebook Save and name notebook Paste in necessary code
Ctrl Shift P Python: Select Interpreter use Python version in ./.venv/bin/python
pip freeze > requirements.txt
pip install -r requirements.txt
pima-indians-diabetes.csv
Extension: Excel Viewer - for viewing csv files in VSCode
per above Python:Select Interpreter 3.10.9 (.venv)
after running pip install ipykernel on running LinearRegression_HandsOn-1.ipynb message appears saying: it is necessary to install ipykernel OK installing ipykernel Rerun LinearRegression_HandsOn-1.ipynb
after running pip install pandas pandas not found
create new jupyter notebook using Ctrl Shift P Create New Jupyter Notebook
- summary-income.md
- high level summary of steps in income.ipynb notebook
jupyter-test jupyter-repo-2 jupyter-3
- LMS - Hands_on_Notebook_Week3.ipynb
- LMS - ENews_Express_Learner_Notebook%5BLow_Code_Version%5D.ipynb
- LMS - abtest.csv
- 2.13 Pandas - Accessing and Modifying DataFrames (condition-based indexing)
- Google Colab mount drive
- search string: pause git tracking
- Git source control in VS Code
Windows Anaconda conda create --name .cenv y conda activate .cenv
python3
not installed so Windows store opens install Python 3.10
python3 -m venv .venv
command was slow at first but self-resolved
- search string: stuck on $ python3 -m venv .venv setting up environment in virtaulenv using python3 stuck on ...
- search string: installing collected packages stuck why is the pip install process stuck on ''Installing collected packages" step?
- search string: np.clip
- numpy.clip | numpy.org | Documentation
- search string: np broadcast against dataframe python
- Q/ What does the term "broadcasting" mean in Pandas documentation? | A/ the term broadcasting comes from numpy | stackoverflow
- broadcasting examples in pandas documentaton | linked to by previous reference on broadcasting | pandas.org | Documentation
- Broadcasting | definition: | The term broadcasting describes how NumPy treats arrays with different shapes during arithmetic operations. | NumPy | numpy.org | Documentation
- Universal functions (ufunc) | NumPy | numpy.org | Documentation
- EDA: from is_categorical def | Check if dataframe column is Categorical | print(is_categorical(data[col])) | stackoverflow
- pandas.get_dummies | Pandas | pandas.pydata.org | Documentation
- search string: pd.get_dummies
- search string: plotting fig from subplot returns Figure(1500x1000)
- fig, ax = plt.subplots()
- matplotlib docs fig, ax = plt.subplots()
- search string: subplot
- matplotlib.pyplot.subplot
- search string: fig.patch.set_facecolor('xkcd:blue')
- xkcd.com/color/rgb/
- search string: fig, axs = plt.subplots(2, 2)
- Creating multiple subplots using plt.subplots >> Stacking subplots in two directions
- matplotlib.pyplot.hist
- search string matplotlib.pyplot histogram
- Histogram with Boxplot above in Python
- search string histogram_boxplot matplotlib
- search string Non-default argument follows default argumentPylance
- SyntaxError: non-default argument follows default argument
- statsmodels.stats.proportion.proportions_ztest
- search string: what are model predictors statsmodels
- Prediction (out of sample) | statsmodels.org | statsmodels | Documentation
- search string: sklearn
- scikit-learn | Machine Learning in Python
- Getting Started -- skikit-learn
- Citing scikit-learn
- User Guide
- Installing scikit-learn
- Scikit-learn: Machine Learning in Python Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.
- redirects to https://scikit-learn.org/stable/ (link 2 in this section, above) Source code, binaries, and documentation
- search string: ipykernel
- pip install ipykernel ipykernel 6.19.2
-
search string: pandas plot frame color -matplotlib
-
search string: pandas plot
-
search string pd crosstab
-
ANSWER to color: seaborn.set_style()
-
search string: sns seaborn color frame facecolor
-
search string: sns.set_theme(style="whitegrid")
-
search string: countplot sns perc
-
search string: pandas facecolor
-
Recursively looping through an object to build a property list | stackoverflow
-
search string: how to recursively return all levels of an object
-
search string: is matplotlib. pyplot an object?
-
search string: matplotlib pyplot plt
-
search string: pd.crosstab color
-
saved search string: (autocomplete) [pd.crosstab df normalize='index').plot(kind="bar", figsize=(6,8),stacked=True)](link) -Creating Links in Markdown
-
[deprecated] | matplotlib.pyplot.figure | matplotlib | Documentation
-
search string: ply.figure frame color
-
search string: countplot | recursively unpacck ax in sns countplot
-
search string: countplot sns ax frame
-
search string: countplot sns
- find in page: | your path PermissionError: [Errno 13] Permission denied | terminal error trying to install preinstalled .venv | stackoverflow
- search string: Error: [Errno 13] Permission denied: 'C:\Users\OneDrive\Documents\.venv\Scripts\python.exe'
- to remove with 0 Axes: comment line: | plt.figure(facecolor='blue').set_facecolor('xkcd:cerulean blue') I used matplotlib, but the error message '<Figure size 720x576 with 0 Axes>' appeared with graph
- fig.savefig('asdf.png')
- How to save the Pandas dataframe from pd.crosstab as a figure (with render_mpl_table)? | stackoverflow
- search string: pd.crosstab "set_facecolor"
- ResidentMario / missingno | Issue | Matplotlib error: 'AxesSubplot' object has no attribute 'set_facecolor' #25 | GitHub
- search string: AttributeError: 'DataFrame' object has no attribute 'set_facecolor'
- scikit-learn/scikit-learn | Issue| 'AxesSubplot' object has no attribute 'set_axis_bgcolor' #10762 | GitHub
- search string: AttributeError: 'DataFrame' object has no attribute 'set_axis_bgcolor'
main.py (files 1 to 4) and script.sh in CoderSales/machine-learning-classification (repository reference below)
- repository reference CoderSales/machine-learning-classification
- slice strings in python
- Check if Python Package is installed
- pip install notebook
- How to Execute Shell Commands with Python
- How to print a string literally in Python
- 4 ways to add variables or values into Python strings
- search string: percentage symbol pip bash
- search string: python access "Option -c 4"
- How to Execute Shell Commands with Python
- import subprocess | subprocess.run('/path/to/script.sh', check=True) os.system() | run all shell commands with a single call
- used for first attempt at naming arbitrary number of variables
- second attempt at naming arbitrary number of variables
- Remove name, dtype from pandas output of dataframe or series
- 2ndary source for turning off index on pandas dataframe print out
-
String Into Variable Name in Python Using the vars() Function
-
option used Python Template String Formatting Method
- Python dictionary append: How to add Key-value Pair?
- Python Dictionary update() Method
- EDA: def is_categorical | Update dictionary items with a for loop | stackoverflow
- detailed confusion matrix Precision and recall
- used for calculation of F1 score Harmonic mean
- image Geometric proof without words that max (a,b) > root mean square (RMS) or quadratic mean (QM) > arithmetic mean (AM) > geometric mean (GM) > harmonic mean (HM) > min (a,b) of two distinct positive numbers a and b
- image QM_AM_GM_HM_inequality_visual_proof.svg/2560px-QM_AM_GM_HM_inequality_visual_proof.svg.png
- Search string: F-beta score
- Search string: F-beta score is the weighted harmonic mean of precision and recall
- Search string: f2 ml sklearn
- fbeta_score sklearn.metrics.fbeta_score
- fbeta_score sklearn.metrics.fbeta_score
- Keyboard Shortcuts > workbench.action.duplicateWorkspaceInNewWindow Ctrl Shift Alt N (modified from suggested on site) VSCODE workflow window views
- not used box-shadow: red
- used change body tag background color behind image
- search string: css font color
- CSS Text
- search string: add color using nb black
- bar How to change color in markdown cells ipython/jupyter notebook? | stackoverflow
- not used to crop images in css
- harmonic mean .svg file
- harmonic mean .svg file page 2
- means visual proof
- How to edit color via code of svg file with: open svg file in explorer > inspect element > Elements > edit circle tag fill attribute
- import subprocess Python: How to script virtual environment building and activation?
- Put this in main.py: | import yoursubfile | Treat it like a module: import file.How can I make one python file run another? [duplicate] | Get one python file to run another, using python 2.7.3 and Ubuntu 12.10:
-
The error is pretty clear. The file hello.py is not an executable file. You need to specify the executable: subprocess.call(['python.exe', 'hello.py', 'htmlfilename.htm']) OSError: [WinError 193] %1 is not a valid Win32 application
-
Using the subprocess Module | python 3.11.2 subprocess — Subprocess management | Using the subprocess Module | python 3.11.2
-
Your best option would be to do it in a function
-
activate () { . ../.env/bin/activate} How to source virtualenv activate in a Bash script
-
def my_function(): Python Functions
-
Main result: If you want to ignore a file that you've committed in the past, you'll need to delete the file from your repository and then add a .gitignore rule for it. | search string: how to add files to gitignore
-
JavaScript function definition syntax (uses curly brackets like bash syntax)Function.prototype.apply()
-
search string: '.' is not recognized as an internal or external command,
-
5 Ways to Fix the "Not Recognized as an Internal or External Command" Error in Windows
-
search string: subprocess.Popen() documentation
- Python Virtual Environments: A Primer
- How Does a Virtual Environment Work?
- The VIRTUAL_ENV environment variable is only available if the virtual environment is activated. For instance: $ python3 -m venv myapp $ source myapp/bin/activate (myapp) $ python -c "import os; print(os.environ['VIRTUAL_ENV'])" /path/to/virtualenv/myapp
- What is the difference between executing a Bash script vs sourcing it?
- How to activate python virtualenv through shell script?
- search string: run python3 -m venv from shell script
- search string: [Error: [Errno 13] Permission denied: 'C:\Users](https://www.google.com/search?q=Error%3A+%5BErrno+13%5D+Permission+denied%3A+%27C%3A%5C%5CUsers%5C&newwindow=1&sxsrf=AJOqlzWirByzwKUkHddqGQt9p_WYB-8a3Q%3A1676448744989&ei=6JPsY66GPOGx8gK9_6_YCQ&ved=0ahUKEwiumrefipf9AhXhmFwKHb3_C5sQ4dUDCA8&uact=5&oq=Error%3A+%5BErrno+13%5D+Permission+denied%3A+%27C%3A%5C%5CUsers%5C&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzoKCAAQRxDWBBCwA0oECEEYAEoECEYYAFDsBVjsBWCTCWgBcAF4AIABN4gBN5IBATGYAQCgAQHIAQjAAQE&sclient=gws-wiz-serp)
- search string: chmod executable shell script
- chmod +x Steps to write and execute a script
- search string: how to start shell script
- search string: run shell using source
- The first line in Bash scripts is a character sequence known as the "shebang." The shebang is the program loader's first instruction when executing the file, and the characters indicate which interpreter to run when reading the script. | Add the following line to the file to indicate the use of the Bash interpreter: | #!/bin/bash How to Write a Bash Script with Examples | Writing a Bash Script | Adding the "shebang" | #!/usr/bin/env | Uses the env program to locate the interpreter. Use this shebang for other scripting languages, such as Perl, Python, etc.
- search string: what does comment do at top of shell script
- How to activate a Python virtual environment from a script file
- search string: python file to start venv
- search string: pass raw strings from shell or py file to terminal to run command in terminal
- search string: Taking Linux Command as Raw String in Python
- Taking Linux Command as Raw String in Python
- search string: how to pass raw code to terminal
- What are some ways to pass raw bytes to a program via the Linux terminal?
- Pass bash argument to python script
- search string: pass arg to function python through bash call
- venv — Creation of virtual environments | An example of extending EnvBuilder
- search string: try catch shell python venv
- PermissionError: [Errno 13] Permission denied
- How to assign the output of a Bash command to a variable? [duplicate]
- search string: #!/bin/bash -x PWD=
pwd - search string: how to activate venv in existing shell
- search string: use shell to activate venv
- Learn X in Y minutes
- Writing shell scripts
- search string: automate virtual env