| Sorry about not doing the reply-all - many decades of trying not to do that accidentally …
Agree on /usr/local - we are making the same point but two different ways. On any unix type system any developer simply has to take care to avoid problems with /user/local - “ignore at your peril” if you like … Certainly the professional projects I have led and/or been involved with have always taken serious steps to obviate these problems. Most of the time a chroot gaol was the answer - hence my fondness for containers which are a more full-fat version of the same thing. Interestingly even “dedicated build servers” ended up becoming infected with cruft … leading back to container-like build systems.
Agreed on sandbox-exec - I mentioned it just so you could see what some are thinking about. There are some challenges for gaols on macOS in recent versions given the development of SIP - where system dyld’s are not present in the user visible filesystem.
Best Paul
Please remember to copy the list on all replies.
Of course one can't rely on something being in /usr/local, but that misses the point: If you install something in /usr/local the compiler is going to find it unless there's another instance in the command line search list (-I and -L). Worse, /usr/local/bin is on the default path. That's why Homebrew-on-Intel's default of creating symlinks there changes Homebrew from a convenience to a serious vulnerability. Poorly controlled package managers like that are a rich opportunity for bad guys, e.g. https://arstechnica.com/information-technology/2021/11/malware-downloaded-from-pypi-41000-times-was-surprisingly-stealthy/.
The sandbox-exec article is interesting, though I'm skeptical that Apple is really using it for much: It's built on the Application Sandbox feature that's required for App Store and Notarized apps. *That* part of it is obviously not going away, but it looks like Apple wants us to use Launch Services for accessing it, not the original sandbox_init(). One could set up a jhbuild environment that way, but it's probably no less complicated that the chroot jail method that most Unix developers already know how to do and that Apple can't disable in the future.
Regards, John Ralls
You’d be surprised just how much 1980’s (and earlier) code is still running - hence my now-and-again use of 3270 ... In my case, I do talks for kids on “how things used to be” and so keeping roughly familiar with MVS and some other systems is a important to me.
You will see that I used the English passive-aggressive “friendly” when talking about CMake - of course, it’s anything but that!
On /usr/local though, no application being built for distribution should ever rely on anything in that subtree - unless the application developer puts the code there as part of an installation it can’t rely on it being there. Such is life on unix … I can’t count the number of times I have seen people accidentally link to something in /usr/local only to find that their application won’t load or breaks in some horrible way on a different machine.
On the “build in a container” front, I totally agree - though to build on MacOS you don’t want a docker container. Docker would give you a linux environment … As an aside, many like podman as a docker alternative, but that is best installed using, er, *cough* *cough*, homebrew!
Best Paul
3270 is still a thing? Scary.
CMake's bug isn't really that recent, the fink and MacPorts inclusion goes back more than 10 years, and homebrew-on-intel uses /usr/local that's on the hard-coded defaults built into the kernel and every Unix compiler I've met. This is also the first time I've heard anyone claim that Cmake is friendly!
Meson's pretty nice. The only wart I've run up against is that fallback feature, which is fine if your environment happens to fit the fallback's settings but is otherwise a PITA. A lot of the GNOME projects have migrated their builds to it.
I'll think a bit on how to word the paragraph in Building Gtk-OSX. I think the ultimate solution is Docker-style containers but since Docker is very much not Free I'm reluctant to promote it. Flatpak would be an obvious GNOME alternative but AFAIK it doesn't play well with macOS and I don't have any bandwidth available to do anything about that.
Regards, John Ralls
I guess it all depends on whether you use your Mac exclusively for developing gtk apps or not. Certainly on my Macs, I have need of a number of tools that are out of-date in the Apple supplied versions and so use homebrew for more recent versions. In the possibly most simple use case, I use htop to monitor system performance - Apple does not supply it, so I use homebrew … I use meld, supplied again by homebrew … I also develop a number of different projects where a unix-like set of tools are required. If I want to develop linux stuff for linux, I have machines or VMs for that ...
Oh, I also use applications such as c3270 (part of x3270) for accessing some legacy systems … I get them from homebrew ..
So it goes on. The gtk-osx world and the rest-of-the-world should really find a way to live with each other :-) I think you’re more or less there though ...
Now, there are a couple of projects that I am working on that use gtk-osx and one in particular that is trying to push using jhbuild. This is why we are in email contact with each other in the first place!
Yes, CMake’s bug is recent - but then Arm-based Macs are relatively new too.
I totally agree about autotools being horrific, but would advance the idea that CMake is a sinner too. Cmake is yet another build tool that attempts to be too clever or “friendly” and causes mayhem - in my case, it messed up with building freetype. There will no doubt be other issues that I come across in due course.
Thanks for the heads up on ignoring the paths … I’ll stick that in my jhbuild-custom and let you know how it goes.
Sorry for the misdirect on meson - I guess I had misread the log … and I’ve not seen meson before.
I looked at your wiki page - I think you could even be a bit stronger with your advice especially as the “use a different user” doesn’t put it any more? Something like:
“If you have homebrew, macports or fink installed on your system, you need to take special steps to ensure that these do not interfere with gtk-osx. When running gtk-osx-setup.sh and whenever you use jhbuild, you need to ensure that no environment variables reference any of the paths used by these tools. You also need to ensure that your jhbuild-local includes the following:
cmakeargs = '-DCMAKE_SYSTEM_IGNORE_PATH="/opt/homebrew:/opt/macports:/sw:/usr/local"' “
WIth those instructions, anyone should be able to work out what to do I think.
Meanwhile thanks for all the help !
Best Paul
I consider it normal to *not* have homebrew, fink, and MacPorts installed. Why would anyone want to make their Mac behave like Linux? Contrariwise, if you want to develop for one of those environments what use do you have for Gtk-OSX?
cmakeargs = '-DCMAKE_SYSTEM_IGNORE_PATH="/opt/homebrew:/opt/macports:/sw"'
I've added that instruction to the Building Gtk-OSX wiki page.
AFAICT meson does *not* have this problem as long as you don't let it do its fallback trick on a Crake-based dependency, and the way to prevent that is to make sure that all fallback dependencies are already built the way you want them.
Regards, John Ralls
As mentioned, I’m on an M1 Mac where everything homebrew does is in /opt/homebrew … there is no /usr/local “pollution” from homebrew on M1 Macs … I agree that there may have been problems with the x86 version where homebrew used /usr/local as its own fiefdom! Now, as I said earlier in the thread, there is nothing at all in /usr/local/lib. There is no environment member containing ‘homebrew’.
Looking at the situation with freetype, the simple fact is that something in /opt should not and does not affect another packages or build systems. The reasoning here is that package A is required to ignore package B when package B is in /opt - this is a fundamental principle of unix filesystems for the past few decades ! The upshot from this is that it's down to package A to not look at or notice package B. If package A is that fussy, it has problems and won’t do well. Whatever package A thinks, it has no option but to work around the fact that /opt/B exists ...
The freetype build seems to be a problem - and it appears this is caused by cmake putting the paths for fink, macports and homebrew into its search paths for include files and libraries. If you look in:
<your gtk installation path>/inst/share/cmake-3.20/Modules/Platform/Darwin.cmake
… you will see …
— snip -- if(CMAKE_SYSTEM_NAME STREQUAL "Darwin" AND CMAKE_SYSTEM_PROCESSOR STREQUAL "arm64") list(PREPEND CMAKE_SYSTEM_PREFIX_PATH /opt/homebrew # Brew on Apple Silicon ) endif() — snip —
This means that all CMake invocations by meson on M1 Macs will include /opt/homebrew in the search path for include and libs. This is wrong - very, very wrong.
The upshot of this is that to use your script as-is requires a special M1 Mac with no homebrew, fink or macports on it - your advice/requirement to “build using a different username” simply does not work reliably! That’s not a sensible situation really is it?
Now, there is some traffic in the Cmake repo regarding a new flag that can be sent to CMake to force it to omit some paths. Given that you want to keep all the gtk stuff separate, this would mean telling CMake to not search /opt/homebrew and /usr/local (at least - not check for fink/macports properly). This would result in you being able to insulate your builds from whatever else is installed on the Mac - you would simply include the system library paths in CMake and omit everything else except within jhbuild’s world. That would be sustainable …
I think that when this version of CMake is released, it would make sense for meson to pass the CMAKE_SYSTEM_IGNORE_PREFIX_PATH flag to CMake. Presumably this could be done by jhbuild - but let’s see?
Meanwhile, thinking about gtk-osx-setup.sh, I think you could be missing an opportunity - I think that on M1 Macs at least, you are perhaps being over restrictive / prescriptive about homebrew. As things stand your script happily ignores /opt/homebrew provided that homebrew is not in any environment variables. Your script could even clean up the environment so that jhbuild is not ‘polluted’ - this is very simple to do. In fact, the script is doing what it is required to do to be a decent citizen, so I have no problem with it - especially now that we have shaken down a couple of bugs. Once we have the meson/cmake bug sorted out, your script and homebrew can peacefully coexist.
It could be that by embracing the fact that many or most developers need tools from homebrew that you could attract more people to using gtk on MacOS? Perhaps one way would be to have an example jhbuild-custom that removes homebrew paths and perhaps sorts the cmake issue on M1 ?
If nothing happens, I expect you will get more comments from M1 based Mac developers ...
Best Paul
On .pythonversion, got it. Glad that's fixed.
On brotlidec in homebrew are you *sure* that there's no link to it in /usr/local/lib? That's one of Homebrew's major malignancies, by default it links its files into the default search locations. I'm not inclined to add work-arounds for users who ignore my precondition that homebrew should not be installed. You can add the additional argument to jhbuild-custom. That file exists specifically to customize your build.
Regards, John Ralls
Happy to be of some help!
My issue with ~/.pythonversion was due to my rough fix for the pip issue:
—snip— export PYTHON_CONFIGURE_OPTS="--enable-shared" export PYENV_PYTHON_VERSION=3.10.0 $PYENV install -v $PYENV_PYTHON_VERSION PIP="$PYENV_ROOT/shims/pip" $PYENV local 3.10.0 $PIP install --upgrade --user pip — snip —
The use of "pyenv local 3.10.0” creates the ~/.pythonversion file - but using the *cough* correct pip3, there’s no need for a pipenv before pip3 … so no .pythonversion :-)
I did a clean run with your latest pushed changes and confirm that the setup script now works.
These problems are just like Pascal’s problems with "ft-fb.h not found” last month - but I did make some progress.
The key issue is that the freetype builds, despite being run in an environment where no environment variables contain the word “homebrew”, still decides to find the librotlidec library in /opt/homebrew/lib. This causes a build failure later on ...
Adding "-D CMAKE_DISABLE_FIND_PACKAGE_BrotliDec=TRUE” to both steps in GtkApplication-osx.modules resolves this issue. This looks like a bug on freetype's part - they should not be scanning random parts of the filesystem ! Anyhow, when disabling the search for this library, and re-running, I end up with all steps of the instructions on the webpage working.
Perhaps you want to make a change GTK-OSX.modules, perhaps not. The big thing here is that developers should not have to have a separate username, vm or machine to build GtkApplication apps - especially when some simple configuration can resolve these issues. Simply put for many, the tools that macports or homebrew provide are required part of daily workflows. Maybe putting a catch at the start of gtk-osx-setup.sh would be useful? Something like:
—snip-- CHECK_HOMEBREW=`printenv |grep homebrew`
If [[ Z$CHECK_HOMEBREW == “Z” ]] ; then echo “Can’t use this script with a homebrew environment !” exit 1 fi — snip --
I’ll get around to trying to build the simple app that I started out on two weeks ago - and will report back if this ends up being problematic.
Thanks for your help meanwhile.
Regards Paul
Ah, got it. Fix pushed. Thanks. I also finally figured out the cause of "pyenv: pip: command not found" and pushed a fix for that too. BTW, I don't see ~/.pythonversion. Are you sure that's from pyenv? Regards, John Ralls On Feb 15, 2022, at 2:08 PM, Spock <spock rogersfamily me uk> wrote:
Sorry - I did not explain myself well enough there John.
I guess the issue that I’m pointing out is that gtk-osx-setup.sh writes the jhbuild script but writes an absolute path for $PATH to be set to rather than escaping so that $PATH is read from the environment when running jhbuild.
In the code snip from gtk-osx-setup.sh below the second $PATH should be escaped to be ‘\$PATH’ (as shown). Without this, it means that PATH only gets set correctly if the environment $PATH when calling gtk-osx-setup.sh contains something like “~/.new_local/bin”.
— snip ---
cat <<EOF > “$DEVPREFIX/bin/jhbuild"
#!$DEVPREFIX/bin/bash
...
export PATH="$PYENV_ROOT/shims:$PATH” <<<<<<<- Should be …/shims:\$PATH
...
exec $DEVPREFIX/bin/pipenv run jhbuild \$@
EOF
— snip ---
This explains why people with fresh machines or fresh usernames end up with the build failing.
So the either the instructions need to say “Set your PATH to include ~/.new_local/bin before invoking gtk-osx-setup.sh” or the script needs to be fixed to include the line:
—snip—
export PATH="$PYENV_ROOT/shims:$PATH”
—snip --
Once I make this change, I can build a lot more of the "jhbuild build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3“ stage.
It still fails, btw, but I need to diagnose why and or ask some more questions!
Hope this makes sense and is helpful?
Regards
Paul
On 15 Feb 2022, at 18:52, john <jralls ceridwen us> wrote:
Well, it's been a requirement since forever that you put $DEVPREFIX/bin in your path before invoking jhbuild, but I suppose there's no harm in also adding it with jhbuild to prevent path pollution. I hadn't noticed ~/.pythonversion. That's rude of them, I'll look for a way to bury that somewhere so that it doesn't affect other stuff.
Regards,
John Ralls
On Feb 15, 2022, at 5:26 AM, Spock <spock rogersfamily me uk> wrote:
Hi John,
I guess the issue with the $PYENV local 23.10.0 is that it causes ~/.pythonversion to be created - which may or may not be a permanent change a developer might want?
The more serious issue is with meson not finding ninja. I think this is down to jhbuild not having ~/.new_local/bin in its path.
Here’s an example:
— snip —
j@pauls-mbp ~ [nobrew] % echo $PATH
/Users/j/.new_local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/Users/j/.cargo/bin
j@pauls-mbp ~ [nobrew] % jhbuild shell
Loading .env environment variables...
Found Command Line Tools 'version: 13.2.0.0.1.1638488800'
Command Line Tools version 13.200000
Prefix: /Users/j/gtk/inst
Entered jhbuild shell, type 'exit' to return.
j@pauls-mbp ~ % jhbuild
zsh: command not found: jhbuild
j@pauls-mbp ~ %
— snip —
I think this means that the jhbuild configuration installed by gtk-osx-setup.sh does not set up jhbuild so that it uses the binaries that have been installed?
I’m not sure if the issue is with jhbuild or gtk-osx-setup.sh - but I guess it’s right to start in this list before looking at what might be wrong (if anything) with jhbuild?
Regards
Paul Rogers
On 15 Feb 2022, at 01:55, John Ralls <jralls ceridwen us> wrote:
On Feb 14, 2022, at 10:03 AM, Spock <spock rogersfamily me uk> wrote:
Hi, I’m running gtk-osx-setup.sh on an M1 Mac/MacOS 12.2 and am having a couple of problems ...
First off, I’m running without any homebrew paths in any of the environment variables.
Now, with this shell, when I run the script, I get the following:
— snip ----
pyenv: pip: command not found
The `pip' command exists in these Python versions:
3.10.0
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
pyenv: pip: command not found
The `pip' command exists in these Python versions:
3.10.0
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
—- snip ---
So I add a line to the script to allow it to find python:
— snip —
PIP=“$PYENV_ROOT/shims/pip”
# Point pyenv at the 3.10.0 Python ...
$PYENV local 3.10.0
$PIP install --upgrade --user pip
— snip ---
With this line, I remove the artefacts from the previous build and re-run. This time the script completes, so I move on to “./.new_local/bin/jhbuild bootstrap-gtk-osx which appears to complete successfully.
*** Was this the right thing to do?
The final step “jhbuild meta-gtk-osx-bootstrap meta-gtk-osx-gtk3 fails due to problems finding a version of ninja …
— snip ---
gtk-doc 1.33.1
User defined options
libdir : lib
prefix : /Users/j/gtk/inst
wrap_mode : nofallback
tests : false
yelp_manual: false
ERROR: Could not detect Ninja v1.8.2 or newer
— snip ---
I checked the version installed by the script …
— snip ---
j@pauls-mbp ~ [nobrew] % ./.new_local/bin/ninja —version
1.10.2
— snip ---
*** So what is happening with ninja? Is there a setup step I’m missing?
Any help much appreciated!
Your fix for pip seems reasonable. Did you remember to add ~/.new_local/bin to $PATH?
Regards,
John Ralls
|
