Otniel ben Kenaz, who was the first Judge of the Jewish People, married Kalev's daughter. I read in one place that he was a half brother to Kalev, and somewhere else it says he was a nephew to Kalev. Which one is right?
tl;dr: The relevant verses are ambiguous; they could be describing Otniel as Kalev's younger brother or as his nephew. However, classical Jewish commentators write that Otniel was Kalev's brother.
עָתְנִיאֵל בֶּן־קְנַז אֲחִי כָלֵב
Otniel, son of Kenaz, brother of Kalev,
עָתְנִיאֵל בֶּן־קְנַז אֲחִי כָלֵב הַקָּטֹן מִמֶּנּוּ
Otniel, son of Kenaz, younger brother of Kalev.
Now, this is ambiguous. Is Otniel the younger brother of Kalev, or is it Kenaz who is Kalev’s brother (which would mean that Otniel was Kalev's nephew)?
One would be tempted to understand that Kenaz is the brother of Kalev, because otherwise we are left with a question: If Kalev and Otniel are brothers, then what was their father's name, Yephuneh or Kenaz?
However, the classical Jewish commentators (unanimously as far as I can tell) do understand that Kalev and Otniel were brothers.
Radak there writes that Kalev and Otniel were full brothers. He suggests that Yephuneh and Kenaz were the same man, with two names. Alternatively, he offers the interpretation (which he prefers) that Kalev and Otniel's father was named Yephuneh, and that Kenaz was a distant ancestor whose name was adopted by his descendants.
Why don't any commentators offer the seemingly more simple solution that Kenaz was Kalev's brother, rather than Otniel?
I believe that this is because Kalev himself is referred to as a Kenizzite (הקנזי) (e.g. in Bamidbar 32:12).
If Kenaz was Kalev's younger brother, this would be a strange way to refer to him. However, if Kenaz was Kalev's step-father, another name for Kalev's father Yephuneh, or the name of one of Kalev's distinguished ancestors, it make a lot more sense.
There's not much more to add to @JoelK's very comprehensive answer.
But it is worth noting that there is a Gemarah in Sotah 11b which also sheds light on his family lineage.
The Gemarah there reads:
(דברי הימים א ב, יח) וכלב בן חצרון הוליד את עזובה אשה ואת יריעות ואלה בניה ישר ושובב וארדון בן חצרון (במדבר יג, ו) בן יפנה הוא בן שפנה מעצת מרגלים
The Gemara discusses the family of Caleb: In Chronicles it says: “And Caleb, the son of Hezron, begot children of Azubah his wife, and of Jerioth, and these were her sons: Jesher, and Shobab, and Ardon” (I Chronicles 2:18). The Gemara asks: Was Caleb actually the son of Hezron? Wasn’t he the son of Jephunneh, as the verse states in Numbers 13:6? The Gemara answers: He was the son of Hezron, but he is called “son of Jephunneh” as an appellation indicating that he was a son who turned away [sheppana] from the counsel of the spies.
ואכתי בן קנז הוא דכתיב (שופטים א, יג) וילכדה עתניאל בן קנז אחי כלב אמר רבא חורגו דקנז הוה
The Gemara asks: But it is still difficult. Hezron could not be his father, as Caleb was the son of Kenaz, as it is written: “And Othniel, the son of Kenaz, Caleb’s younger brother, took it” (Judges 1:13). This would mean that Caleb was also a son of Kenaz. Rava said: Caleb was the stepson of Kenaz, as he and Othniel shared a mother but had different fathers.
(Translation and additional notation from Sefaria)